[sbase] Command list

[sin]

Hi all,

I've been thinking, besides the commands that are already listed in TODO
and all the unimplemented options for existing commands, what other commands
do we want to include in sbase?

The current TODO list includes:

diff, expr, printf, tr, unexpand

What about:

od, logname, uuencode, uudecode, mktemp, passwd, su, sed, patch

Let me know what you guys think.

I think all of these can be implemented portably, otherwise, they are a
candidate for ubase.

Thanks,
sin

[Roberto E. Vargas Caballero]

Post by sin
od, logname, uuencode, uudecode, mktemp, passwd, su, sed, patch

ed and awk are also missed (yeah, I know they are bigger and
more difficult programs, but they are basic for a system).
--
Roberto E. Vargas Caballero
_______________________________________________________________________
'Write programs that do one thing and do it well. Write programs to work
together. Write programs to handle text streams, because that is a
universal interface'

(Doug McIlroy)

In Other Words - Don't design like polkit or systemd
_______________________________________________________________________

[Alex Pilon]

Post by sin
I've been thinking, besides the commands that are already listed in TODO
and all the unimplemented options for existing commands, what other commands
do we want to include in sbase?
[
]
od, logname, uuencode, uudecode, mktemp, passwd, su, sed, patch

Don't od's and hexdump's functionality overlap?

Do we still really need uuencode and uudecode?

Regards,

Alex Pilon

[Thorsten Glaser]

Post by Alex Pilon
Don't od's and hexdump's functionality overlap?

Yes, they (and hd) are normally one binary.

Post by Alex Pilon
Do we still really need uuencode and uudecode?

Yes!

I often use them, in combination with GNU screen, to
copy/paste files(!) from one tab to the other, when
using other methods (such as rsync or scp/sftp) is
unpractical or impossible (think serial console).

bye,
//mirabilos
--=20
<Natureshadow> Boah, ich steck hier soviel Anstrengung
ins Furzen, da=C3=9F ich dabei Schwindelanf=C3=A4lle krieg

[Alex Pilon]

Post by Thorsten Glaser

Post by Alex Pilon
Do we still really need uuencode and uudecode?

Yes!
I often use them, in combination with GNU screen, to
copy/paste files(!) from one tab to the other, when
using other methods (such as rsync or scp/sftp) is
unpractical or impossible (think serial console).

I know the trick. I used base64 for that, not uuencode. Don't they both have the
same overhead, other than uuencode's header? That header's redundant anyway, at
least for this hack, since you can just use tar or cpio.

Regards,

Alex Pilon

[Thorsten Glaser]

Post by Alex Pilon
I know the trick. I used base64 for that, not uuencode. Don't they
both have the same overhead, other than uuencode's header?

Yes but uuencode is more portable=E2=80=A6 e.g. the Linux =E2=80=9Cbase64=
=E2=80=9D tool is
rarely installed anywhere and is extremely picky about the formatting
of its input; OpenBSD has b64decode but it demands header lines similar
to uuencode (MirBSD has b64decode -r which is not picky either =E2=80=93 bu=
t
that=E2=80=99s only in MirBSD of course).

Nowadays, uuencode (=E2=80=9Csharutils=E2=80=9D) is not installed everywher=
e any more
by default, but it=E2=80=99s still the most widespread portable one and usu=
ally
easy to get if it=E2=80=99s not yet there.

Meh, maybe I should write a shell uu{en,de}code implementation and
put it into the standard .mkshrc I ship=E2=80=A6

bye,
//mirabilos
--=20
This space for rent.

[Truls Becken]

A few more suggestions:

bc dc dd file find fmt install killall less make sh time which xargs
(un)zip/gzip/bzip2/lzma/xz

Some of these are more complex than others of course.

-Truls

[Chris Down]

sh

There have already been a few discussions about the sbase shell, I believe the
last opinion given was that we shouldn't include one.

which

This should be left to the shell usually, since almost everyone uses a hash
table for lookups nowadays, and we have no idea about what's in it or how it
caches.

[sin]

Post by Truls Becken
bc dc dd file find fmt install killall less make sh time which xargs
(un)zip/gzip/bzip2/lzma/xz

dd: I suspect this might not be portable if we want an optimized version.
find: Useless, just do `du -a | grep blabla'
killall: Not portable, candidate for ubase, not sure we even need it.
make: Do we really need this in sbase?
sh: We agreed to do without a shell.
xargs: Why?
(un)zip: Why?
gzip/bzip2/lzma/xz: Maybe as separate programs and not part of sbase?

Thanks,
sin

[Roberto E. Vargas Caballero]

Post by sin
dd: I suspect this might not be portable if we want an optimized version.
find: Useless, just do `du -a | grep blabla'

and a 'find -type d -name xxxx -prune -o -type f -perm g=x -exec chmod g-x {} \; '?
find is a mini language for run over directories, and it is a must.

Post by sin
xargs: Why?

if you have a directory with a very, very, very, very large number of files and
then you try a 'rm *' you will see why you need xargs. I use it a lot in combination
with find. For example in the previous example a better way (faster):

find -type d -name xxxx -prune -o -type f -perm g=x | xargs chmod g-x

Regards,
--
Roberto E. Vargas Caballero
_______________________________________________________________________
'Write programs that do one thing and do it well. Write programs to work
together. Write programs to handle text streams, because that is a
universal interface'

(Doug McIlroy)

In Other Words - Don't design like polkit or systemd
_______________________________________________________________________

[sin]

Post by Roberto E. Vargas Caballero

Post by sin
dd: I suspect this might not be portable if we want an optimized version.
find: Useless, just do `du -a | grep blabla'

and a 'find -type d -name xxxx -prune -o -type f -perm g=x -exec chmod g-x {} \; '?
find is a mini language for run over directories, and it is a must.

Not UNIX.

Post by Roberto E. Vargas Caballero

Post by sin
xargs: Why?

if you have a directory with a very, very, very, very large number of files and
then you try a 'rm *' you will see why you need xargs. I use it a lot in combination

I realize when xargs is useful, I just hate it.

[Roberto E. Vargas Caballero]

Post by sin

Post by Roberto E. Vargas Caballero

Post by sin
dd: I suspect this might not be portable if we want an optimized version.
find: Useless, just do `du -a | grep blabla'

and a 'find -type d -name xxxx -prune -o -type f -perm g=x -exec chmod g-x {} \; '?
find is a mini language for run over directories, and it is a must.

Not UNIX.

Not UNIX?, what do you mean?. Sorry, I don't understand what you want say here.
--
Roberto E. Vargas Caballero
_______________________________________________________________________
'Write programs that do one thing and do it well. Write programs to work
together. Write programs to handle text streams, because that is a
universal interface'

(Doug McIlroy)

In Other Words - Don't design like polkit or systemd
_______________________________________________________________________

[sin]

Post by Roberto E. Vargas Caballero

Post by sin

Post by Roberto E. Vargas Caballero

Post by sin
dd: I suspect this might not be portable if we want an optimized version.
find: Useless, just do `du -a | grep blabla'

and a 'find -type d -name xxxx -prune -o -type f -perm g=x -exec chmod g-x {} \; '?
find is a mini language for run over directories, and it is a must.

Not UNIX.

Not UNIX?, what do you mean?. Sorry, I don't understand what you want say here.

I do not consider find(1) to be elegant in any way. I agree many ppl use it,
including myself, I just wish we could do without this madness.

[Raphaël Proust]

Post by sin
I do not consider find(1) to be elegant in any way. I agree many ppl use it,
including myself, I just wish we could do without this madness.

Maybe we need a way to filter lists of file names:
ls | ff -type d | ff -name xxxx


Seems more compositional… Unixier one might say.

[Roberto E. Vargas Caballero]

Post by sin
I do not consider find(1) to be elegant in any way. I agree many ppl use it,
including myself, I just wish we could do without this madness.

Well, we can not agree in all the points ;). I see find like the best example of
the eleganty of Unix, do one thing and do it well. Other programs should not
run over directories, and the best example of this problem is tar, which
needs a lot of different options in order to select the correct files, instead
of doing it like cpio, which reads the input list of files from stdin, so
you can use a highly specialized tool for it, that it is of course find.

Regards,
--
Roberto E. Vargas Caballero
_______________________________________________________________________
'Write programs that do one thing and do it well. Write programs to work
together. Write programs to handle text streams, because that is a
universal interface'

(Doug McIlroy)

In Other Words - Don't design like polkit or systemd
_______________________________________________________________________

[sin]

Post by Roberto E. Vargas Caballero

Post by sin
I do not consider find(1) to be elegant in any way. I agree many ppl use it,
including myself, I just wish we could do without this madness.

Well, we can not agree in all the points ;). I see find like the best example of
the eleganty of Unix, do one thing and do it well. Other programs should not
run over directories, and the best example of this problem is tar, which
needs a lot of different options in order to select the correct files, instead
of doing it like cpio, which reads the input list of files from stdin, so
you can use a highly specialized tool for it, that it is of course find.

As this seems to upset the sbase audience, a small sensible subset of the find(1)
functionality can be implemented for sbase I suppose.

[Markus Wichmann]

Post by sin
I do not consider find(1) to be elegant in any way. I agree many ppl use it,
including myself, I just wish we could do without this madness.

I can do without it! I just do

setopt extendedglob
chmod g-x **/*(#q.E^AI)~xxxx/**/*

(I hope I understood that find command correctly)

The only possible prolem would arise from the argumen list being too
long, and then I can just

zmodload zsh/files

which implements chmod inside of zsh. Or I could

print -l **/*(#q.E^AI)~xxxx/**/* | xargs chmod g-x

But of course that requires the zsh. So there's at least something all
that bloated code buys.

Ciao,
Markus

[Alexander S.]

Contributing into the discussion about find.
Most of its selectors are implemented by stest(1); commonly useful
ones that I can think of are -user and -group. However:
1) ostracize me, but I like find "one dash, long option" style way
more than stest "a thousand of letters a-la test". I understand the
test(1) legacy here, I just say that it wouldn't be very pleasant to
consult with manual each time I'm trying to write a command.
2) there is no way to express intersection, union and negation of a
test. They are mostly irrelevant for test(1), but here we ought to
have at least disjunction, because making temporary lists and merging
them with sort -u or comm(1) is unpleasant to say the least. (Also,
slow). Theoretically speaking, every logical expression can be
represented in conjunctive normal form, so providing support for
negation and union and using multiple commands & pipes as intersection
("| stest test1 | stest test2") should be enough, but in some cases,
unpleasant too.
3) we need a program that can print a directory tree well, with
-mindepth, -maxdepth, and sorted/unsorted options.
4) we need some standard way to separate file names in a stream.
Basically, ^@ is our only refuge, because path can contain any other
character. But maybe newline is good enough?
5) we need a program that can take input and convert into a
commandline. So, xargs (but of course without space-separation
quotation nonsense. Not once nor twice had I written the glorious "tr
'\n' '\0' | xargs -0" to turn it off). I suppose most of POSIX options
are worth keeping.

So, the former "find" command would be like
ptree -maxdepth 2 | ftest "( perm 111 || newer ./reference ) && user
root" | xargs rm
I feel that going the awk way (e. g. inventing the mini-language and
passing expression as one string) is better than find way, with -a and
-o and escaped parens.

[Alex Pilon]

Post by Alexander S.
4) we need some standard way to separate file names in a stream.
character. But maybe newline is good enough?

Why not just play it safe? Would it really add any noticeable
complexity, even if just an option?

Post by Alexander S.
Not once nor twice had I written the glorious "tr '\n' '\0' | xargs -0"
to turn it off).

Which does nothing for safety. `xargs -d '\n'` would have been just as
good.

Post by Alexander S.
ptree -maxdepth 2 | ftest "( perm 111 || newer ./reference ) && user root" | xargs rm

The Solaris and SunOS users *might* not like you for that choice of name
(“ptree”), but who cares? It's inconsequential.

Regards,

Alex Pilon

[Chris Down]

Post by Alex Pilon

Post by Alexander S.
4) we need some standard way to separate file names in a stream.
character. But maybe newline is good enough?

Why not just play it safe? Would it really add any noticeable
complexity, even if just an option?

In my opinion, having no way to write secure programs using these tools
due to this would undermine all the great work being done in sbase.

[Truls Becken]

Post by sin
find: Useless, just do `du -a | grep blabla'

I'm not interested in disk usage, but finding files based on certain
properties, such as update time, ownership, permissions, etc.

Post by sin
make: Do we really need this in sbase?

Make is a basic tool, useful in lots of situations.
It would be nice not to rely on GNU for this.

Post by sin
sh: We agreed to do without a shell.

Yes, and I agree for the user shell. It would still be nice to have a
minimalistic one for scripts, where extensions should be banned anyway.

Post by sin
xargs: Why?

I use it all the time, most frequently to call less or grep on a list of
files produced by some other command.

-Truls

[Thorsten Glaser]

Post by Truls Becken
bc dc

bc can be done on top of dc; BSD does that (the dc uses OpenSSL
for arbitrary-precision numbers).

Post by Truls Becken
killall

I object, killall should never, ever, be used. (Try it on a
Solaris system, for example.)

Post by Truls Becken

Post by sin
make: Do we really need this in sbase?

Make is a basic tool, useful in lots of situations.

I=E2=80=99d call it a development tool, normally.

Post by Truls Becken
It would be nice not to rely on GNU for this.

NetBSD make is portable (they use it for pkgsrc=C2=AE).
(MirBSD=E2=80=99s is, somewhat, but in a sucky way, and
I=E2=80=99ve not had time to redo it since I learned a
lot in the meantime; it was only a quick hack,
never planned to persist, anyway.)

Post by Truls Becken

Post by sin
sh: We agreed to do without a shell.

Yes, and I agree for the user shell. It would still be nice to have a
minimalistic one for scripts, where extensions should be banned anyway.

The problem is that all existing =E2=80=9Cminimalistic=E2=80=9D ones suck.
The posh approach (take pdksh and strip all extensions
away) is interesting, but posh is very buggy, and the
author said he=E2=80=99d base on mksh were he to restart now.
But maintainance effort is hell on this (and you *will*
accidentally remove things that are not extensions, and
you *will* run into the problem that POSIX is, slowly,
standardising most of them like $((=E2=80=A6)) and $'=E2=80=A6' now).

That being said, mksh/lksh makes a nice /bin/sh for most systems=E2=80=A6

bye,
//mirabilos
--=20
This space for rent.

[Strake]

Post by sin
find: Useless, just do `du -a | grep blabla'

Yes, this is what one does on Plan 9.

Post by sin
I realize when xargs is useful, I just hate it.

Yep, the basic function is sane, but the other crap, insert mode and
quotation and such, loses.

Post by sin

Post by sin
find: Useless, just do `du -a | grep blabla'

I'm not interested in disk usage, but finding files based on certain
properties, such as update time, ownership, permissions, etc.

du -a | cut -f 2

Cheers,
Strake

[Truls Becken]

Post by Strake

Post by Truls Becken
I'm not interested in disk usage, but finding files based on certain
properties, such as update time, ownership, permissions, etc.

du -a | cut -f 2

Yes, but how do you filter the result on some combination of file attributes?

du -a | cut -f2 | xargs ls -dl | grep <awful pattern> | sed 's/.*:[0-9]* //' ?

And then you would have a hard time specifying pattern for "newer than 5 days".

If you want to do it this way around, a command is missing for filtering a
list of file names, as already mentioned by Raphaël. It could be generalised
to a command for filtering stdin by calling a predicate command, e.g:

du -a | cut -f2 | filter test -w

For this, test is missing "newer than 5 days" since that's what find does.

-Truls

[Truls Becken]

Post by Truls Becken
If you want to do it this way around, a command is missing for filtering a
list of file names, as already mentioned by Raphaël. It could be generalised
du -a | cut -f2 | filter test -w

Just for kicks I wrote filter. With this, the above gives same result as:

find . -perm +222

At least after sorting both outputs, and as long as the fact that these
particular tests do not check the same thing does not bite you. Not sure if it
is useful. It is certainly not standard. As already mentioned, test does not
provide the same set of tests as find.

-Truls

[Evan Gates]

Post by Truls Becken
find . -perm +222

Is this not what stest does?

-emg

[Evan Gates]

Post by Truls Becken
du -a | cut -f2 | filter test -w

I also find it worth mentioning that newline delimited lists of
filenames are not safe as newlines are legal in filenames. This also
disregards the possibility of tabs in filenames (and many other
solutions run into problems with whitespace as well).

find -exec and shell globs are good solutions as they don't depend on
printing and then reading a list of files. find -print0 may be
acceptable, but is not portable. xargs runs into many of the same
problems. (If you want to be POSIX compatible and feel that you must
print and read paths, try find ././ and use "././" as a separator. But
man is that gross.)

I vaguely recall reading about a set of tools that someone was writing
(I think it was cls?) that solved many of these problems by using
environment variables RS and FS or similar (like awk) to format their
output. Seeing as sbase is not aiming for POSIX compliance perhaps
something along these lines is worth considering?

-emg

[Alex Pilon]

Post by Strake

Post by sin
find: Useless, just do `du -a | grep blabla'

Yes, this is what one does on Plan 9.

Which doesn't necessarily make it right.

Post by Strake

Post by sin
I realize when xargs is useful, I just hate it.

Yep, the basic function is sane, but the other crap, insert mode and
quotation and such, loses.

So just don't implement the insane bits! Does anybody here use anything
but ‘-d’ or ‘-0’ and/or ‘-L’?

Post by Strake

Post by sin
I'm not interested in disk usage, but finding files based on certain
properties, such as update time, ownership, permissions, etc.

du -a | cut -f 2

No. ‘du -0 | cut -f 2-’, at least.

And what the heck? That has nothing to do with what *you* just quoted above.

We're talking about checking *update time*, *ownership*, *permissions*,
etc., not the file name. This means I have to wrap in shell, and before
I know it, my little traversal of a few million files (not kidding, real
use case, though it's noticeable for smaller numbers) takes longer, not
to mention that it's more code.

du -a \
| cut -f 2- \
| while read -rd f; do
if (($(stat --format=%a "$f") & 0111)) &&
[ $(stat --format=%u "$f") = $(id -u) ] &&
[ "$f" -nt $reference ] &&
[ -f "$f" ]; then
chmod a-x "$f"
fi
done

Still have to figure out how to make it work with ‘du -a0’, without
using the following.

awk 'BEGIN { RS="\0" } { sub("^[0-9]+[[:space:]]*", "", $0); print $0 }'

I've heard that the above is non-portable due to the use of nul as the
record separator. Should we care? It neither works with nawk or Plan 9
awk.

You can cut a line or two if in the previous while loop if you make it
slightly less readable but the point still stands, whereas


find -print0 -perm /111 -uid $(id -u) -type f -cnewer $reference | xargs -0 rm

is short and sweet.

Please don't implement ‘-delete’. It's both redundant and not in POSIX.
Are there any other considerations that would make us want it?

du's output is standardized
(http://pubs.opengroup.org/onlinepubs/009604499/utilities/du.html#tag_04_40_10),
mercifully, though that doesn't mean it makes handling a few special
cases easy. Should we care about any such? The only ones that I can
think of right now are newlines in the file names, which is pretty
stupid but possible.

Using du is brittle but you can use it without fear the overwhelming
majority of the time. Still, if you advocate strong typing as a Haskell
programmer, why wouldn't you in shell (well, not shell itself) as
appropriate?

Regards,

Alex Pilon

[Alex Pilon]

Apologies for the spam, some typos here
 though you likely knew what I
meant.

Post by Alex Pilon
No. ‘du -0 | cut -f 2-’, at least.

`du -a 
`

Post by Alex Pilon
du -a \
| cut -f 2- \
| while read -rd f; do

`while read -rd '' f; do`

Still not portable outside of bash and mksh due to the use of `-d`, to
the detriment of busybox or d?ash users..

Regards,

Alex Pilon

[Simon Lieb]

Hi,

Post by sin
Hi all,
I've been thinking, besides the commands that are already listed in TODO
and all the unimplemented options for existing commands, what other commands
do we want to include in sbase?

May sbase be interested in a small utility called « slow » I wrote some
time ago to slow down down input to output ? I can make a patch for
sbase.

I mainly use it with ii and to avoid trigger anti‐flood system and some
other cases.

See my git repository[0].

Regards,


[0]: git:://home.tibu.fr/slow

[Simon Lieb]

Mmh, I should have attached an archive.

Regards,

[Chris Down]

Post by Simon Lieb
May sbase be interested in a small utility called « slow » I wrote some
time ago to slow down down input to output ? I can make a patch for
sbase.
I mainly use it with ii and to avoid trigger anti‐flood system and some
other cases.

In my opinion, this is too niche for sbase. Thanks for the code, though,
although I mostly use pv -L (and yes, pv sucks).

[Simon Lieb]

Post by Chris Down

Post by Simon Lieb
May sbase be interested in a small utility called « slow » I wrote some
time ago to slow down down input to output ? I can make a patch for
sbase.
I mainly use it with ii and to avoid trigger anti‐flood system and some
other cases.

In my opinion, this is too niche for sbase. Thanks for the code, though,
although I mostly use pv -L (and yes, pv sucks).

I understand. Didn't know about pv, thanks for the hint.

[Louis Santillan]

Late to the party, but in response to Truls email request on
10/18/2013, there's some less sucky lossless compression/decompression
code at <https://code.google.com/p/miniz/> (Zip/deflate & zlib
compatible, small, fast) and <https://code.google.com/p/lzham/>
(nearly LZMA compression with much faster decompression, Windows only
for the time being :/).

-Louis