copy-pasting directories using the shell

Posted by: admin  :  Category: Debian GNU/Linux, FreeBSD, HowTo's, OS X, RHEL, Shells

Imagine that you need to copy over some files or directories to another host, but the security policy or the connectivity doesn’t allow to use standard file transfer protocols. Here’s a quick and dirty solution to overcome such restrictions.

This little trick involves the ability to access two different hosts via SSH (or telnet, or even a serial console) and a terminal client supporting to capture screen output into a file or a copy-paste buffer.

So, on the source host, go for the file/directory, and tar it up, apply some compression to it as needed, send the output to stdout and pipe it directly through base64:

tar -czpvf - some/path | base64

This will give you some output and the base64-encoded representation of the data, which may look like this:

some/path
some/path/somefile1
some/path/somefile2
some/path/somefile3
some/path/somedir
some/path/somedir/someotherfile1
H4sIAGnAGVgAA+3POw7CQAwFwD1KbkC8kfY+kUIJK+XD+VklDVBAQ7qZ5lnyK+zp+rhs83if6i2d
pW9KKXs2n7nPkVsjIuccqY8hSqRuOO2iF9uyjnPXpbnW9Vvv1/74I46M+O+RAAAAAAAAAAAA8O4J
FJs7gwAoAAA=

The “garbage” shown after the file and directory names is the base64 encoded contents from tar.
Don’t bother decoding the output above, this is just some gargabe from /dev/urandom to illustrate this example 😉

Now, copy-paste just the base64 output (or send it to a file, if your terminal client supports this).
Then, on the supposed-to-be target host, change to the directoy, where your files/directories should end up, then emter the command below:

cat|base64 -d|tar -xzpvf -

Don’t worry, it’ll “hang” on an empty line.
Now paste the buffer (or send the contents of the file captured before into the buffer).

This will looks similar to this:

cat|base64 -d|tar -xzpvf -
H4sIAGnAGVgAA+3POw7CQAwFwD1KbkC8kfY+kUIJK+XD+VklDVBAQ7qZ5lnyK+zp+rhs83if6i2d
pW9KKXs2n7nPkVsjIuccqY8hSqRuOO2iF9uyjnPXpbnW9Vvv1/74I46M+O+RAAAAAAAAAAAA8O4J
FJs7gwAoAAA=

As soon as the buffer is flushed, output will string “hang”, press CTRL-D to complete the transactions.

If done correctly, the input should be sent trough base64 to be decoded, and then passed on to tar to unpack.
You should see the file and directory names accordingly.

cat|base64 -d|tar -xzpvf -
H4sIAGnAGVgAA+3POw7CQAwFwD1KbkC8kfY+kUIJK+XD+VklDVBAQ7qZ5lnyK+zp+rhs83if6i2d
pW9KKXs2n7nPkVsjIuccqY8hSqRuOO2iF9uyjnPXpbnW9Vvv1/74I46M+O+RAAAAAAAAAAAA8O4J
FJs7gwAoAAA=
some/path
some/path/somefile1
some/path/somefile2
some/path/somefile3
some/path/somedir
some/path/somedir/someotherfile1

That’s it, a while directory tree copied without involging file transer protocols.

Of course, the base64 encoding adds some overhead, so this doesn’t work well for huge data loads as it’s limited to the console speed. However this is a very quick solution if only a few files need to be copied quickly without bothering about possible restrictions.

MySQL replication: Cross-Database updates using Procedures and dynamically-switched row-based Replication

Posted by: admin  :  Category: MySQL, Programming

What a lengthy title to describe what this is about. Dough!

Well, I wanted to solve an interesting problem: Imagine a multi-master MySQL cluster, where databases are only fully replicated between masters, but only partially between connected stub slaves.

Now, assume there is two databases, one is called DB1, the other is called DB2.
Both master servers carry both databases, while the connected slaves get only DB2 (enforced by a replicate-wild-do-table=DB2.% config statement in the slave’s my.cnf config file).

+-------+      +--------+      +--------+      +-------+
| SLAVE |  | MASTER |  | MASTER |  | SLAVE |
+-------+      +--------+      +--------+      +-------+
 only DB2         all             all           only DB2
 database       databases       databases      databases

Now, let’s image a query, which populates entries in DB2 by selecting data from DB1.

INSERT INTO DB2.destTable1 SELECT * FROM DB1.sourceTable1;

The way how MySQL replication works, this query is about to fail on the slaves because the referenced source table does not exist.
Of course, this is depending heavily on the actual configuration of the master, and how the log updates in the binary log are handled.

If statement-based replication is enabled, the replication will totally blow up.
On the other hand, if row-based replication is chosen, it is likely to complete.

Why? Because for statement-based replication, the actual statement is replicated as shown above.
With row-based replication, the master will, while writing the logs, expand the statement into the actual actions carried out, so the exact result can be passed on to the slaves.

Let’s see what this looks like:

SET INSERT_ID=1734227546/*!*/;
INSERT INTO destTable1 (field1, field2, field3, field4) VALUES ('val1', 'val2', 'val3', 'val4');

The advantage: The query can be actually carried out on the slave even if the referred to source Table would not exist.
This is was row-based replication is about.

Now, there is a third option for MySQL, which is mixed-mode replication. Actually, in more recent versions of MySQL, this is the default.
In mixed-mode, MySQL will choose by itself, if and under what conditions, statement-based or row-based replication applies.

This is actually a good default in most cases and I use it on all my setup. But in said scenario above, mixed-mode replication will cause unexpected replication failure.

So, it would be fine if there is a way to runtime-switch the replication mode, wouldn’t it?
Luckily there is.

One may either do this globally, or more feasible in this case, per-session.
So first force MySQL into row-based replication before executing the actual query, and then revert-back to mixed-mode replication afterwards after the query has been run.

set session binlog_format = 'ROW';
INSERT INTO DB2.destTable1 SELECT * FROM DB1.sourceTable1;
set session binlog_format = 'MIXED';

This can even be used within a stored procedure. So I can infact add a stored procedure on DB1, which carries out changes to DB2.

CREATE DEFINER=`root@localhost` PROCEDURE `procSampleProcedure`()
    MODIFIES SQL DATA
BEGIN
set session binlog_format = 'ROW';
INSERT INTO DB2.destTable1 SELECT * FROM DB1.sourceTable1;
set session binlog_format = 'MIXED';
END

This is very nice indeed as I can still run MySQL in mixed-mode by default, and only force it into RBR (row-based replication) when needed.
Essentially, this works around potential replication lockups due to missing dependant tables.

Remoting an old APC PDU using SNMP and remOcular

Posted by: admin  :  Category: Hardware, Perl, Programming, Utilities

Garage sale at the office: Good chance to grab on some (very) old hardware, like an APC 9221 PDU. Yes, it’s old (some 15 years or so), so surely not state of the art. But yet good enough to use in my home lab. Who could tell that there were some unforeseen issues waiting for me …
Read more…

Ansible in 10 minutes or less

Posted by: admin  :  Category: Debian GNU/Linux, FreeBSD, HowTo's, Operating Systems, Scripting

I just remember a recent argument I had with someone about automation. It’s unbelievable, how many things are still done manually on a widespread scale, not leveraging the possibilities at all. Especially with so many frameworks available to help out, sticking to “the old way” ain’t just cool any more.

So let’s quickly look at Ansible, and how we can be up and running for even simple task automation in 10 minutes or less.
Read more…

quick-and-dirty PAM with LUA, mod_magnet and lighttpd -or- how to breach system security

Posted by: admin  :  Category: Programming, RegExp, Security, Shells

Be warned: This example serves as an illustration on how to *NOT* do it.
It’s just one of my examples I teach to apprentices at the office when it comes to learning scrips, and how important data input validation (or the absence of the same) is.
It’s also a good illustration on how attackers may break into systems to steal data or make them part of a botnet.

The given situation depicts a lighttpd server, which exposes a directory which must be protected via LDAP-managed accounts, so there is an immediate need for PAM. However, lighttpd lacks a PAM implementation. Period. There’s a very ugly and highly insecure way however …

Read more…

FreeBSD on ARMv6: Cross-Compile Performance Optimization for Poudriere

Posted by: gdelmatto  :  Category: FreeBSD, Operating Systems, Programming

Important Announcements on FreeBSD-armv6 packages

While initially writing this article, I had the idea to establish a service where packages can be selected to build for armv6. As of February 2016 this service is now online.
If you just need current FreeBSD packages for armv6, this is the place to visit. Otherwise, keep on reading.

Whilst playing around with FreeBSD on Raspberry Pi, I started to dig into cross-compiling packages.

Well, if you follow the first tutorial you’ll surely notice that there is no real speed-gain, because the use of full binary emulation on a x86 host through QEMU. So this is almost as slow as if packages were natively compiled on the Raspberry Pi itself even if done on a multi-cpu Xeon powerhouse.

So let’s see how to get an actually performance gain.

Read more…

Transform Cobalt Raq3 into a Raspberry Pi-powered Media Center

Posted by: gdelmatto  :  Category: Debian GNU/Linux, Hacks, Hardware, Operating Systems, Programming, Scripting

Anyone remember these adorable blueish 1U servers made by Cobalt Networks?

ppcobaltraq

While I was never in true love with the Cobalt OS itself, I actually liked the Cobalts Raq enclosure.
So much that I salvaged one while cleaning out a data center last summer. I decided to grant it a second live as a media center box running OSMC.
And of course it’s powered by a Respberry Pi. Nowadays there’s simply no way around those nice little boxes 😉
Read more…

Bash Script to rip CD/DVD ISO image on OS X

Posted by: admin  :  Category: Operating Systems, OS X, Programming, Scripting

Apple’s OS X has an easy way to rip a CD/DVD image using Disk Utility program.
However, you’ll end up with a file in that is not in ISO format, thus utterly useless if you want to re-use the file for virtualization purposes or on another operating system unable to handle those .cdr files.

For a one-shot option, OS X provides everything to convert the .cdr to .iso files, which is outlined at http://imacify.com/2013/06/how-to-create-iso-disc-image-from-cddvd-in-mac-os-x/.

If you do however plan to rip a lot (and I mean, a lot!) of CDs/DVDs to ISO files on OS X, here’s a little bash script I came up with.

Read more…

Inofficial FreeBSD port for Zend Optimizer Plus

Posted by: gdelmatto  :  Category: FreeBSD, PHP

2017-03-12: A US-based company claims that their trademark is infringed by mentioning the terms “(Zend) Optimizer+ / (Zend) Optimizer Plus” on this page. In context of my writings, the use of these terms are clearly related to the well-known open source software by Rogue Wave Software (formerly Zend Technologies), as released in 2013.

While the contents of this page has become somewhat obsolete, as Zend Optimizer+/Opcache has since been integrated into the PHP programming language itself, the content is left here for reference purposes.

Again: I make it absolutely clear that this content exists as a documentation only in the scope if IT terminology and  a well-known open source software product Zend Optimizer (former trademark).

The use of the term specifically relates to the original product name, including file name references. As a reference, the original commit to the source is linked here.

“Zend Optimizer+” has been rebranded to “Zend Opcache” in the meantime.


While just in the process of doing web-server freshup on FreeBSD, I was caught by the good news that Zend Technologies have released their Zend opcode caching engine as open source.

Now it’s called Zend Optimizer+ and hosted over there at github.

As far as I have seen, it did not yet popup as a buildable port on FreeBSD’s ports tree, but that can only take little time for today.
So I quickly made up my own port which you can download here.

To build it, simply download and extract the file to /usr/ports/devel:

cd /usr/ports/devel
fetch -o- http://phaq.phunsites.net/files/2013/02/ZendOptimizerPlus.tgz | tar -xzpvf -

Then “make install” as usual:

cd /usr/ports/devel/ZendOptimizerPlus
make install

Afterwards, running “php -i” (or phpinfo from a web-accessible script file) should denote it runs “with Zend Optimizer+ 7.0.0-dev”.

Done 🙂

Nagios/Icinga Plugin to check for DokuWiki Updates (v2,2015-09-05)

Posted by: gdelmatto  :  Category: HowTo's, Perl, Programming, Scripting

Nagios/Icinga can also serve to send you friendly reminders, like for example that you need to perform software updates.

Here’s my little contribution, a simple plugin to monitor a given DokuWiki site and check against the release server for any upgrades.

Just fetch the  check_dokuwiki-0.1 check_dokuwiki-0.2 tarball and extract the check_dokuwiki script to your Nagios/Icinga plugin directory.

The latest plugin update (0.2, 2015-09-06) catches up with latest updates on Dokuwikis web page, and now incorporates upstream checks using the same mechanism that Dokuwiki uses internally. Also it adds some (although not yet well tested) support for http authentication.

Please check out the cli help for more details on specific arguments:

check_dokuwiki v (nagios-plugins 1.4.15)
The nagios plugins come with ABSOLUTELY NO WARRANTY. You may redistribute
copies of the plugins under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING.
Copyright (c) 2012 Gianpaolo Del Matto

Usage:
  check_dokuwiki -H 
  check_dokuwiki [-h | --help]
  check_dokuwiki [-V | --version]

more arguments
  -a | --use-auth              enable http authentication mode
  -r name | --realm=name       use given realm with http auth
  -u user | --username=user    use given username for http auth
  -p pass | --password=pass    use given password for http auth
  -s | --use-ssl               enable SSL mode (uses TCP:443 as default, see --tcp-port
  -p num | --tcp-port num      use non-standard port
                               if not given, defaults to TCP:80 (or TCP:443 if --use-ssl is used)

   	give any valid DokuWiki hostname to fetch the 'VERSION' file from.
                         Note: HTTP AUTH is currently not supported.

Send email to nagios-users@lists.sourceforge.net if you have questions
regarding use of this software. To submit patches or suggest improvements,
send email to nagiosplug-devel@lists.sourceforge.net.
Please include version information with all correspondence (when possible,
use output from the --version option of the plugin itself).

Register the plugin with a command definition like this:

# 'check_dokuwiki' command definition
define command{
        command_name	check_dokuwiki
        command_line	/usr/local/libexec/nagios/check_dokuwiki -H $HOSTNAME
	}

Then simply add a service to one or more of your DokuWiki hosts (or hostgroups, whatever you prefer).

define service{
        use                     generic-service
        host_name               your_wiki_host_objects_list_here
        service_description     dokuwiki_version
        check_command           check_dokuwiki
        max_check_attempts      5
        check_interval          5
        retry_interval          3
        check_period            24x7
        notification_interval   0
        notification_period     24x7
        notification_options    w,c,r
        }

Restart Nagios/Icinga and you’re done.

Happy monitoring 🙂