sed

Introduction

sed can quickly filter, clean, and write logs for QA purposes.

sed reads from stdin (standard input) line-by-line, making it powerful for live streaming logs (e.g., adb logcat, nc, logs from protocols like FIX etc).

Basic Syntax

sed [options] commands [input-file]

• [options] — flags to control sed behavior.

• 'command' — what transformation to apply.

• [file...] — input file(s) or piped input.

Common Options

Option	Example	Description
-i	sed -i s/old/new/ file.txt	In-place editing
-n	sed -n /pattern/p file.txt	Suppress automatic printing
-e	sed -e s/old/new/ -e /pattern/d file.txt	Execute multiple commands
-f	sed -f script.sed file.txt	Read commands from a file
-r	sed -r s/old/new/ file.txt	Use extended regular expressions
-E	sed -E s/old/new/ file.txt	Use extended regular expressions (similar to -r)
-z	sed -z s/old/new/ file.txt	Separate lines with NUL character
-l	sed -l 100 l file.txt	Specify the line length for the 'l' command
-b	sed -b s/old/new/ file.txt	Binary mode (do not strip the CR characters)

---

Performance Tips (For Working with Large Logs)

• Combine multiple operations

• Use -n with p for selective output

• Test without -i first

• For huge files, consider splitting

Optimize sed for Speed

• Use -n + p – Suppress automatic printing and explicitly p only needed lines:

sed -n '/error/p' large.log > errors.txt

• Limit scope with addresses – Reduce processing by targeting specific line ranges:

sed '10000,20000s/foo/bar/' large.log

• Avoid backreferences – Capture groups \(\...\) and & replacements slow processing.

Parallel Processing

• GNU Parallel + sed – Process chunks simultaneously:

parallel --pipepart --block 100M -a large.log sed 's/foo/bar/'

• Split & Parallelize – For non-GNU systems:

split -l 1000000 large.log chunk_ && \
parallel -j $(nproc) sed -i 's/foo/bar/' ::: chunk_* && \
cat chunk_* > processed.log

• Multi-file processing – Apply changes to many files concurrently:

find /var/log -name "*.log" | parallel -j 8 sed -i 's/old/new/g'

Memory-Efficient Patterns

• Chain commands – Combine operations in a single sed call:

sed -e 's/foo/bar/' -e '/baz/d' large.log

Prefer d over D – Line deletion (d) is faster than pattern-space manipulation (D). Use q for early exit – Stop processing after finding a match:

sed '/critical error/q' large.log

Pre-Processing Tricks

• Filter first with grep – Reduce data before complex sed operations:

grep "pattern" large.log | sed 's/.../.../'

sed vs Alternatives

Feature	sed	awk	perl
Speed	Fast	Medium	Slow
Complexity	Simple	Medium	Complex
In-place edit	Yes	No	Yes
Regex support	Basic	Full	Full

Conclusion

sed excels at:

• Quick text substitutions

• Automated file editing

• Stream processing

For complex data manipulation, consider awk or perl.

---

📌 Note: In RCA/log viewing, sed is typically used with -n and p commands to extract, highlight, and format, but not modify files.

🧠 Tip: Always prefer to run sed without -i during investigations. Stream output into a safe file or view in terminal.

sed Cheatsheet