copy-pasting directories using the shell

Posted by: admin  :  Category: Debian GNU/Linux, FreeBSD, HowTo's, OS X, RHEL, Shells

Imagine that you need to copy over some files or directories to another host, but the security policy or the connectivity doesn’t allow to use standard file transfer protocols. Here’s a quick and dirty solution to overcome such restrictions.

This little trick involves the ability to access two different hosts via SSH (or telnet, or even a serial console) and a terminal client supporting to capture screen output into a file or a copy-paste buffer.

So, on the source host, go for the file/directory, and tar it up, apply some compression to it as needed, send the output to stdout and pipe it directly through base64:

tar -czpvf - some/path | base64

This will give you some output and the base64-encoded representation of the data, which may look like this:

some/path
some/path/somefile1
some/path/somefile2
some/path/somefile3
some/path/somedir
some/path/somedir/someotherfile1
H4sIAGnAGVgAA+3POw7CQAwFwD1KbkC8kfY+kUIJK+XD+VklDVBAQ7qZ5lnyK+zp+rhs83if6i2d
pW9KKXs2n7nPkVsjIuccqY8hSqRuOO2iF9uyjnPXpbnW9Vvv1/74I46M+O+RAAAAAAAAAAAA8O4J
FJs7gwAoAAA=

The “garbage” shown after the file and directory names is the base64 encoded contents from tar.
Don’t bother decoding the output above, this is just some gargabe from /dev/urandom to illustrate this example 😉

Now, copy-paste just the base64 output (or send it to a file, if your terminal client supports this).
Then, on the supposed-to-be target host, change to the directoy, where your files/directories should end up, then emter the command below:

cat|base64 -d|tar -xzpvf -

Don’t worry, it’ll “hang” on an empty line.
Now paste the buffer (or send the contents of the file captured before into the buffer).

This will looks similar to this:

cat|base64 -d|tar -xzpvf -
H4sIAGnAGVgAA+3POw7CQAwFwD1KbkC8kfY+kUIJK+XD+VklDVBAQ7qZ5lnyK+zp+rhs83if6i2d
pW9KKXs2n7nPkVsjIuccqY8hSqRuOO2iF9uyjnPXpbnW9Vvv1/74I46M+O+RAAAAAAAAAAAA8O4J
FJs7gwAoAAA=

As soon as the buffer is flushed, output will string “hang”, press CTRL-D to complete the transactions.

If done correctly, the input should be sent trough base64 to be decoded, and then passed on to tar to unpack.
You should see the file and directory names accordingly.

cat|base64 -d|tar -xzpvf -
H4sIAGnAGVgAA+3POw7CQAwFwD1KbkC8kfY+kUIJK+XD+VklDVBAQ7qZ5lnyK+zp+rhs83if6i2d
pW9KKXs2n7nPkVsjIuccqY8hSqRuOO2iF9uyjnPXpbnW9Vvv1/74I46M+O+RAAAAAAAAAAAA8O4J
FJs7gwAoAAA=
some/path
some/path/somefile1
some/path/somefile2
some/path/somefile3
some/path/somedir
some/path/somedir/someotherfile1

That’s it, a while directory tree copied without involging file transer protocols.

Of course, the base64 encoding adds some overhead, so this doesn’t work well for huge data loads as it’s limited to the console speed. However this is a very quick solution if only a few files need to be copied quickly without bothering about possible restrictions.

([M|m]ac)?\s?OS(\sX)?: Permit ICMP redirects

Posted by: admin  :  Category: Networking, Operating Systems, OS X

So I was fighting around with that Motorola/Netopia router I’m obliged to use, because the network operator doesn’t allow hooking up a custom device.

Well, it is possible after all, as proven in the past, however, in order to use the SIP gateway of network operator (whereas the login credentials are not provided), the operator-branded router must be used. *sigh*

Here’s a somewhat high-level overview: The clients, which shall connect to the lab, are in the same subnet as the default router #1. The destination for more specific lab routes is router #2, which is in the same subnet.

+-----------+        +------------+       +-----------+       +-----------+       +-----------+
| clients   |  ----  | WiFi/Wired | ----  | router #1 | ----  | router #2 | ----  | LAB stuff |
+-----------+        +------------+       +-----------+       +-----------+       +-----------+
{                            CLIENT SUBNET                          } {      LAB SUBNETS      }

So actually, I could just add the more specific routes to any client, indicating it shall forward through router #2.
However, this is cumbersome. I wond’t want to add these routes on every client.

So I tried hacking them into the Motorola/Netopia router. I had my hard time with that, but only because it’s so silly on overly complicated … 🙁
So far so good, my clients could send ICMP echo requests towards the LAB devices, however, that was as close as I could get.

Not every client was capable in accessing everything in the LAB.

As it turned out, the Motorola/Netopia sends ICMP redirects. It does that because router #2 (a cisco, btw) is reachable via the CLIENT subnet and thus directly reachable by anyone in the same subnet.
However, ICMP redirects are somewhat non-deterministic, as the forwarding is not influenced by the router anymore. I consider it vodoo, which is why I prefer turning it off.

The only problem is that this “Netopia SOC OS” doesn’t have an equivalent to a Cisco-type “no ip redirects”-command.
Well, it’s a Linux after all, so I could turn it off by setting /proc/sys/net/ipv4/conf/*/send_redirects to 0. There is an obscure way to break out from the SOC OS shell and get a unrestricted shell:

ping 127.0.0.2;/bin/busybox telnetd -l/bin/sh -p9999

This would open a root shell on port 9999, from where the kernel setting could be changed. However, since this will get reverted whenever the router reboots due to operator updates, I would need to hack this back in. I don’t like this at all. Please, let me officially retrieve the SIP credentials to hook up my IP phone directly, so I can use a Cisco router. Pretty please!

Well, one day perhaps. Until then, I need to get it working with the least intrusive means of configuration.
So, I can’t replace the router, I can’t learn it to not send redirects.
But, if my clients, ([M|m]ac)?\s?OS(\sX)? in particular, don’t play well with ICMP redirects, let’s force them to do so.

On macOS (man, let’s blatandly change the name one more time!), this can be done via the sysctl command in the Terminal.
Query it like this:

# sudo sysctl net.inet.icmp.drop_redirect
net.inet.icmp.drop_redirect: 1

So macOS indeed drops ICMP redirects by default. Let’s change this:

# sudo sysctl net.inet.icmp.drop_redirect=0
net.inet.icmp.drop_redirect: 1 -> 0

With the new setting, connections started to work right away.

A note of caution: In my opinion ICMP redirects can be a dangerous thing, as they open the door for an attacker to influence the client’s idea of the routing table.
A client should not listen to redirects and always forward traffic towards it’s designated router. Overriding a default setting like this may be ok on a case-by-case basis, but should be strongly inspected und monitored.
If I had the choice, I surely had preferred a permanently applied setting on the router instead.

Ansible in 10 minutes or less

Posted by: admin  :  Category: Debian GNU/Linux, FreeBSD, HowTo's, Operating Systems, Scripting

I just remember a recent argument I had with someone about automation. It’s unbelievable, how many things are still done manually on a widespread scale, not leveraging the possibilities at all. Especially with so many frameworks available to help out, sticking to “the old way” ain’t just cool any more.

So let’s quickly look at Ansible, and how we can be up and running for even simple task automation in 10 minutes or less.
Read more…

armv6 Package Builder for FreeBSD is online

Posted by: admin  :  Category: FreeBSD, HowTo's, Operating Systems, Utilities

There it finally is, The Phunsites Package Builder at http://pkgbuild.phunsites.net/.

Since my initial writings on FreeBSD on Raspberry Pi, I’ve always wanted to have a webservice, where I can just select the port I want and it’ll be packaged up in minutes.
Now, there it is. Fully automated, with a neat and (hopefully) easy to use webinterface.

Check it out. It’s free lemons! 😉

pkgbuild2016

Quick&Dirty FreeBSD on Alix (without PXE boot)

Posted by: gdelmatto  :  Category: FreeBSD, Hardware

It’s been a while since my last post and I’ve been quiet busy writing on my graduation essay.

Meanwhilst my colleague Steven donated me a somewhat dated PC Engines Alix computer. I though to put it to some good use as  a packet generator for my new network playground I’m currently building up.

Funny anectode: While googl’ing around on some docs about Alix computers, I stumled accross his 2009 original post on FreeBSD installs.

But then I read that I’d need to go through config hell for DHCP, PXE, NFS for a one-time install … Oh boy, must be kidding …

Read more…

FreeBSD on ARMv6: Cross-Compile Performance Optimization for Poudriere

Posted by: gdelmatto  :  Category: FreeBSD, Operating Systems, Programming

Important Announcements on FreeBSD-armv6 packages

While initially writing this article, I had the idea to establish a service where packages can be selected to build for armv6. As of February 2016 this service is now online.
If you just need current FreeBSD packages for armv6, this is the place to visit. Otherwise, keep on reading.

Whilst playing around with FreeBSD on Raspberry Pi, I started to dig into cross-compiling packages.

Well, if you follow the first tutorial you’ll surely notice that there is no real speed-gain, because the use of full binary emulation on a x86 host through QEMU. So this is almost as slow as if packages were natively compiled on the Raspberry Pi itself even if done on a multi-cpu Xeon powerhouse.

So let’s see how to get an actually performance gain.

Read more…

Transform Cobalt Raq3 into a Raspberry Pi-powered Media Center

Posted by: gdelmatto  :  Category: Debian GNU/Linux, Hacks, Hardware, Operating Systems, Programming, Scripting

Anyone remember these adorable blueish 1U servers made by Cobalt Networks?

ppcobaltraq

While I was never in true love with the Cobalt OS itself, I actually liked the Cobalts Raq enclosure.
So much that I salvaged one while cleaning out a data center last summer. I decided to grant it a second live as a media center box running OSMC.
And of course it’s powered by a Respberry Pi. Nowadays there’s simply no way around those nice little boxes 😉
Read more…

FreeBSD on the Raspberry Pi – Pt 1: Quick and Dirty Intro (also for Linux-Users)

Posted by: gdelmatto  :  Category: FreeBSD, HowTo's

Tonight I went along to fire up FreeBSD on a Raspberry Pi.
Although I’m in for Linux on a day-to-day basis, my heart truely belongs to the BSDish world.
So why not run FreeBSD on the Pi? Especially since I have an idea in mind to build a very specialized and compact access point for one of my ongoing projects.

A good starting point for this is at https://wiki.freebsd.org/FreeBSD/arm.
Read more…

Bash Script to rip CD/DVD ISO image on OS X

Posted by: admin  :  Category: Operating Systems, OS X, Programming, Scripting

Apple’s OS X has an easy way to rip a CD/DVD image using Disk Utility program.
However, you’ll end up with a file in that is not in ISO format, thus utterly useless if you want to re-use the file for virtualization purposes or on another operating system unable to handle those .cdr files.

For a one-shot option, OS X provides everything to convert the .cdr to .iso files, which is outlined at http://imacify.com/2013/06/how-to-create-iso-disc-image-from-cddvd-in-mac-os-x/.

If you do however plan to rip a lot (and I mean, a lot!) of CDs/DVDs to ISO files on OS X, here’s a little bash script I came up with.

Read more…

Inofficial FreeBSD port for Zend Optimizer Plus

Posted by: gdelmatto  :  Category: FreeBSD, PHP

2017-03-12: A US-based company claims that their trademark is infringed by mentioning the terms “(Zend) Optimizer+ / (Zend) Optimizer Plus” on this page. In context of my writings, the use of these terms are clearly related to the well-known open source software by Rogue Wave Software (formerly Zend Technologies), as released in 2013.

While the contents of this page has become somewhat obsolete, as Zend Optimizer+/Opcache has since been integrated into the PHP programming language itself, the content is left here for reference purposes.

Again: I make it absolutely clear that this content exists as a documentation only in the scope if IT terminology and  a well-known open source software product Zend Optimizer (former trademark).

The use of the term specifically relates to the original product name, including file name references. As a reference, the original commit to the source is linked here.

“Zend Optimizer+” has been rebranded to “Zend Opcache” in the meantime.


While just in the process of doing web-server freshup on FreeBSD, I was caught by the good news that Zend Technologies have released their Zend opcode caching engine as open source.

Now it’s called Zend Optimizer+ and hosted over there at github.

As far as I have seen, it did not yet popup as a buildable port on FreeBSD’s ports tree, but that can only take little time for today.
So I quickly made up my own port which you can download here.

To build it, simply download and extract the file to /usr/ports/devel:

cd /usr/ports/devel
fetch -o- http://phaq.phunsites.net/files/2013/02/ZendOptimizerPlus.tgz | tar -xzpvf -

Then “make install” as usual:

cd /usr/ports/devel/ZendOptimizerPlus
make install

Afterwards, running “php -i” (or phpinfo from a web-accessible script file) should denote it runs “with Zend Optimizer+ 7.0.0-dev”.

Done 🙂