hacksKiBi’s bloghttp://mraw.org/blog/tags/hacks/KiBi’s blogikiwiki2019-08-17T06:11:30ZSending HTML messages with Net::XMPP (Perl)http://mraw.org/blog/2019/08/17/Sending_HTML_messages_with_Net_XMPP/2019-08-17T06:11:30Z2019-08-17T06:15:00Z
<h3>Executive summary</h3>
<p>It’s perfectly possible! <a href="http://mraw.org/blog/tags/hacks/#html-demo">Jump to the HTML demo!</a></p>
<h3>Longer version</h3>
<p>This started with a very simple need: wanting to improve the
notifications I’m receiving from various sources. Those include:</p>
<ul>
<li>changes or failures reported during <a href="https://puppet.com">Puppet</a>
runs on my own infrastructure, and on at a customer’s;</li>
<li>build failures for the <a href="https://d-i.debian.org/daily-images/daily-build-overview.html">Debian Installer</a>;</li>
<li>changes in banking amounts;</li>
<li>and lately: build status for jobs in a customer’s Jenkins instance.</li>
</ul>
<p>I’ve been using plaintext notifications for a number of years but I
decided to try and pimp them a little by adding some colors.</p>
<p>While the XMPP-sending details are usually hidden in a local module,
here’s a small self-contained example: connecting to a server, sending
credentials, and then sending a message to someone else. Of course,
one might want to tweak the <code>Configuration</code> section before trying to run
this script…</p>
<div class="highlight-perl"><pre class="hl"><span class="hl slc">#!/usr/bin/perl</span>
<span class="hl kwa">use</span> strict<span class="hl opt">;</span>
<span class="hl kwa">use</span> warnings<span class="hl opt">;</span>
<span class="hl kwa">use</span> Net<span class="hl opt">::</span>XMPP<span class="hl opt">;</span>
<span class="hl slc"># Configuration:</span>
<span class="hl kwc">my</span> <span class="hl kwb">$hostname</span> <span class="hl opt">=</span> <span class="hl str">'example.org'</span><span class="hl opt">;</span>
<span class="hl kwc">my</span> <span class="hl kwb">$username</span> <span class="hl opt">=</span> <span class="hl str">'bot'</span><span class="hl opt">;</span>
<span class="hl kwc">my</span> <span class="hl kwb">$password</span> <span class="hl opt">=</span> <span class="hl str">'call-me-alan'</span><span class="hl opt">;</span>
<span class="hl kwc">my</span> <span class="hl kwb">$resource</span> <span class="hl opt">=</span> <span class="hl str">'demo'</span><span class="hl opt">;</span>
<span class="hl kwc">my</span> <span class="hl kwb">$recipient</span> <span class="hl opt">=</span> <span class="hl str">'human</span><span class="hl ipl">@example</span><span class="hl str">.org'</span><span class="hl opt">;</span>
<span class="hl slc"># Open connection:</span>
<span class="hl kwc">my</span> <span class="hl kwb">$con</span> <span class="hl opt">=</span> Net<span class="hl opt">::</span>XMPP<span class="hl opt">::</span>Client<span class="hl opt">-></span><span class="hl kwd">new</span><span class="hl opt">();</span>
<span class="hl kwc">my</span> <span class="hl kwb">$status</span> <span class="hl opt">=</span> <span class="hl kwb">$con</span><span class="hl opt">-></span><span class="hl kwd">Connect</span><span class="hl opt">(</span>
hostname <span class="hl opt">=></span> <span class="hl kwb">$hostname</span><span class="hl opt">,</span>
connectiontype <span class="hl opt">=></span> <span class="hl str">'tcpip'</span><span class="hl opt">,</span>
tls <span class="hl opt">=></span> <span class="hl num">1</span><span class="hl opt">,</span>
ssl_ca_path <span class="hl opt">=></span> <span class="hl str">'/etc/ssl/certs'</span><span class="hl opt">,</span>
<span class="hl opt">);</span>
<span class="hl kwc">die</span> <span class="hl str">'XMPP connection failed'</span>
<span class="hl kwa">if</span> <span class="hl opt">!</span> <span class="hl kwc">defined</span><span class="hl opt">(</span><span class="hl kwb">$status</span><span class="hl opt">);</span>
<span class="hl slc"># Log in:</span>
<span class="hl kwc">my</span> <span class="hl kwb">@result</span> <span class="hl opt">=</span> <span class="hl kwb">$con</span><span class="hl opt">-></span><span class="hl kwd">AuthSend</span><span class="hl opt">(</span>
hostname <span class="hl opt">=></span> <span class="hl kwb">$hostname</span><span class="hl opt">,</span>
username <span class="hl opt">=></span> <span class="hl kwb">$username</span><span class="hl opt">,</span>
password <span class="hl opt">=></span> <span class="hl kwb">$password</span><span class="hl opt">,</span>
resource <span class="hl opt">=></span> <span class="hl kwb">$resource</span><span class="hl opt">,</span>
<span class="hl opt">);</span>
<span class="hl kwc">die</span> <span class="hl str">'XMPP authentication failed'</span>
<span class="hl kwa">if</span> <span class="hl kwb">$result</span><span class="hl opt">[</span><span class="hl num">0</span><span class="hl opt">]</span> <span class="hl kwc">ne</span> <span class="hl str">'ok'</span><span class="hl opt">;</span>
<span class="hl slc"># Send plaintext message:</span>
<span class="hl kwc">my</span> <span class="hl kwb">$msg</span> <span class="hl opt">=</span> <span class="hl str">'Hello, World!'</span><span class="hl opt">;</span>
<span class="hl kwc">my</span> <span class="hl kwb">$res</span> <span class="hl opt">=</span> <span class="hl kwb">$con</span><span class="hl opt">-></span><span class="hl kwd">MessageSend</span><span class="hl opt">(</span>
to <span class="hl opt">=></span> <span class="hl kwb">$recipient</span><span class="hl opt">,</span>
body <span class="hl opt">=></span> <span class="hl kwb">$msg</span><span class="hl opt">,</span>
type <span class="hl opt">=></span> <span class="hl str">'chat'</span><span class="hl opt">,</span>
<span class="hl opt">);</span>
<span class="hl kwc">die</span><span class="hl opt">(</span><span class="hl str">'ERROR: XMPP message failed'</span><span class="hl opt">)</span>
<span class="hl kwa">if</span> <span class="hl kwb">$res</span> <span class="hl opt">!=</span> <span class="hl num">0</span><span class="hl opt">;</span>
</pre></div>
<p>For reference, here’s what the XML message looks like in Gajim’s XML
console (on the receiving end):</p>
<div class="highlight-xml"><pre class="hl"><span class="hl kwa"><message</span> type='chat' to='human@example.org' from='bot@example.org/demo'<span class="hl kwa">></span>
<span class="hl kwa"><body></span>Hello, World!<span class="hl kwa"></body></span>
<span class="hl kwa"></message></span>
</pre></div>
<p>Issues start when one tries to send some HTML message, e.g. with the
last part changed to:</p>
<div class="highlight-perl"><pre class="hl"><span class="hl slc"># Send plaintext message:</span>
<span class="hl kwc">my</span> <span class="hl kwb">$msg</span> <span class="hl opt">=</span> <span class="hl str">'This is a <b>failing</b> test'</span><span class="hl opt">;</span>
<span class="hl kwc">my</span> <span class="hl kwb">$res</span> <span class="hl opt">=</span> <span class="hl kwb">$con</span><span class="hl opt">-></span><span class="hl kwd">MessageSend</span><span class="hl opt">(</span>
to <span class="hl opt">=></span> <span class="hl kwb">$recipient</span><span class="hl opt">,</span>
body <span class="hl opt">=></span> <span class="hl kwb">$msg</span><span class="hl opt">,</span>
type <span class="hl opt">=></span> <span class="hl str">'chat'</span><span class="hl opt">,</span>
<span class="hl opt">);</span>
</pre></div>
<p>as that leads to the following message:</p>
<div class="highlight-xml"><pre class="hl"><span class="hl kwa"><message</span> type='chat' to='human@example.org' from='bot@example.org/demo'<span class="hl kwa">></span>
<span class="hl kwa"><body></span>This is a <span class="hl kwd">&lt;</span>b<span class="hl kwd">&gt;</span>failing<span class="hl kwd">&lt;</span>/b<span class="hl kwd">&gt;</span> test<span class="hl kwa"></body></span>
<span class="hl kwa"></message></span>
</pre></div>
<p>So tags are getting encoded and one gets to see the uninterpreted
“HTML code”.</p>
<p>Trying various things to embed that inside <code><body></code> and <code><html></code> tags,
with or without namespaces, led nowhere.</p>
<p>Looking at a message sent from Gajim to Gajim (so that I could craft
an HTML message myself and inspect it), I’ve noticed it goes this way (edited to concentrate on important parts):</p>
<div class="highlight-xml"><pre class="hl"><span class="hl kwa"><message</span> <span class="hl kwb">xmlns</span>=<span class="hl str">"jabber:client"</span> <span class="hl kwb">to</span>=<span class="hl str">"human@example.org/Gajim"</span> <span class="hl kwb">type</span>=<span class="hl str">"chat"</span><span class="hl kwa">></span>
<span class="hl kwa"><body></span>Hello, World!<span class="hl kwa"></body></span>
<span class="hl kwa"><html</span> <span class="hl kwb">xmlns</span>=<span class="hl str">"http://jabber.org/protocol/xhtml-im"</span><span class="hl kwa">></span>
<span class="hl kwa"><body</span> <span class="hl kwb">xmlns</span>=<span class="hl str">"http://www.w3.org/1999/xhtml"</span><span class="hl kwa">></span>
<span class="hl kwa"><p></span>Hello, <span class="hl kwa"><strong></span>World<span class="hl kwa"></strong></span>!<span class="hl kwa"></p></span>
<span class="hl kwa"></body></span>
<span class="hl kwa"></html></span>
<span class="hl kwa"></message></span>
</pre></div>
<p>Two takeaways here:</p>
<ul>
<li><p>The message is send both in plaintext and in HTML. It seems Gajim
archives the plaintext version, as opening the history/logs only
shows the textual version.</p></li>
<li><p>The fact that the HTML message is under a different path
(<code>/message/html</code> as opposed to <code>/message/body</code>) means that one
cannot use the <code>MessageSend</code> method to send HTML messages…</p></li>
</ul>
<p>This was verified by checking the documentation and code of the
<code>Net::XMPP::Message</code> module. It comes with various <em>getters</em> and
<em>setters</em> for attributes. Those are then automatically collected when
the message is serialized into XML (through the <code>GetXML()</code>
method). Trying to add handling for a new HTML attribute would mean being
extra careful as that would need to be treated with <code>$type = 'raw'</code>…</p>
<p>Oh, wait a minute! While using <code>git grep</code> in the sources, looking for
that <code>raw</code> type thing, I’ve discovered what sounded promising: an
<code>InsertRawXML()</code> method, that doesn’t appear anywhere in either the
code or the documentation of the <code>Net::XMPP::Message</code> module.</p>
<p>It’s available, though! Because <code>Net::XMPP::Message</code> is derived from
<code>Net::XMPP::Stanza</code>:</p>
<div class="highlight-perl"><pre class="hl"><span class="hl kwa">use</span> Net<span class="hl opt">::</span>XMPP<span class="hl opt">::</span>Stanza<span class="hl opt">;</span>
<span class="hl kwa">use</span> base <span class="hl kwc">qw</span><span class="hl opt">(</span> Net<span class="hl opt">::</span>XMPP<span class="hl opt">::</span>Stanza <span class="hl opt">);</span>
</pre></div>
<p>which then in turn comes with this function:</p>
<div class="highlight-perl"><pre class="hl"><span class="hl slc">##############################################################################</span>
<span class="hl slc">#</span>
<span class="hl slc"># InsertRawXML - puts the specified string onto the list for raw XML to be</span>
<span class="hl slc"># included in the packet.</span>
<span class="hl slc">#</span>
<span class="hl slc">##############################################################################</span>
</pre></div>
<p>Let’s put that aside for a moment and get back to the <code>MessageSend()</code>
method. It wants parameters that can be passed to the
<code>Net::XMPP::Message</code> <code>SetMessage()</code> method, and here is its entire
code:</p>
<div class="highlight-perl"><pre class="hl"><span class="hl slc">###############################################################################</span>
<span class="hl slc">#</span>
<span class="hl slc"># MessageSend - Takes the same hash that Net::XMPP::Message->SetMessage</span>
<span class="hl slc"># takes and sends the message to the server.</span>
<span class="hl slc">#</span>
<span class="hl slc">###############################################################################</span>
<span class="hl kwa">sub</span> MessageSend
<span class="hl opt">{</span>
<span class="hl kwc">my</span> <span class="hl kwb">$self</span> <span class="hl opt">=</span> <span class="hl kwc">shift</span><span class="hl opt">;</span>
<span class="hl kwc">my</span> <span class="hl kwb">$mess</span> <span class="hl opt">=</span> <span class="hl kwb">$self</span><span class="hl opt">-></span><span class="hl kwd">_message</span><span class="hl opt">();</span>
<span class="hl kwb">$mess</span><span class="hl opt">-></span><span class="hl kwd">SetMessage</span><span class="hl opt">(</span><span class="hl kwb">@_</span><span class="hl opt">);</span>
<span class="hl kwb">$self</span><span class="hl opt">-></span><span class="hl kwd">Send</span><span class="hl opt">(</span><span class="hl kwb">$mess</span><span class="hl opt">);</span>
<span class="hl opt">}</span>
</pre></div>
<p>The first assignment is basically equivalent to <code>my $mess = Net::XMPP::Message->new();</code>, so
what this function does is: creating a <code>Net::XMPP::Message</code> for us, passing all
parameters there, and handing the resulting object over to the <code>Send()</code>
method. All in all, that’s merely a proxy.</p>
<p><a name="html-demo"></a></p>
<h4>HTML demo</h4>
<p>The question becomes: what if we were to create that object ourselves,
then tweaking it a little, and then passing it directly to <code>Send()</code>,
instead of using the slightly limited <code>MessageSend()</code>? Let’s see what
the rewritten sending part would look like:</p>
<div class="highlight-perl"><pre class="hl"><span class="hl slc"># Send HTML message:</span>
<span class="hl kwc">my</span> <span class="hl kwb">$text</span> <span class="hl opt">=</span> <span class="hl str">'This is a working test'</span><span class="hl opt">;</span>
<span class="hl kwc">my</span> <span class="hl kwb">$html</span> <span class="hl opt">=</span> <span class="hl str">'This is a <b>working</b> test'</span><span class="hl opt">;</span>
<span class="hl kwc">my</span> <span class="hl kwb">$message</span> <span class="hl opt">=</span> Net<span class="hl opt">::</span>XMPP<span class="hl opt">::</span>Message<span class="hl opt">-></span><span class="hl kwd">new</span><span class="hl opt">();</span>
<span class="hl kwb">$message</span><span class="hl opt">-></span><span class="hl kwd">SetMessage</span><span class="hl opt">(</span>
to <span class="hl opt">=></span> <span class="hl kwb">$recipient</span><span class="hl opt">,</span>
body <span class="hl opt">=></span> <span class="hl kwb">$text</span><span class="hl opt">,</span>
type <span class="hl opt">=></span> <span class="hl str">'chat'</span><span class="hl opt">,</span>
<span class="hl opt">);</span>
<span class="hl kwb">$message</span><span class="hl opt">-></span><span class="hl kwd">InsertRawXML</span><span class="hl opt">(</span><span class="hl str">"<html><body></span><span class="hl ipl">$html</span><span class="hl str"></body></html>"</span><span class="hl opt">);</span>
<span class="hl kwc">my</span> <span class="hl kwb">$res</span> <span class="hl opt">=</span> <span class="hl kwb">$con</span><span class="hl opt">-></span><span class="hl kwd">Send</span><span class="hl opt">(</span><span class="hl kwb">$message</span><span class="hl opt">);</span>
</pre></div>
<p>And tada!</p>
<div class="highlight-xml"><pre class="hl"><span class="hl kwa"><message</span> type='chat' to='human@example.org' from='bot@example.org/demo'<span class="hl kwa">></span>
<span class="hl kwa"><body></span>This is a working test<span class="hl kwa"></body></span>
<span class="hl kwa"><html></span>
<span class="hl kwa"><body></span>This is a <span class="hl kwa"><b></span>working<span class="hl kwa"></b></span> test<span class="hl kwa"></body></span>
<span class="hl kwa"></html></span>
<span class="hl kwa"></message></span>
</pre></div>
<p>I’m absolutely no expert when it comes to XMPP standards, and one
might need/want to set some more metadata like <code>xmlns</code> but I’m happy
enough with this solution that I thought I’d share it as is. <code>;)</code></p>
Mark a mail as read across maildirshttp://mraw.org/blog/2014/08/11/Mark_a_mail_as_read_across_maildirs/2014-08-11T18:18:33Z2014-08-11T18:20:00Z
<h2>Problem</h2>
<p>Discussions are sometimes started by mailing a few different mailing
lists so that all relevant parties have a chance to be aware of a new
topic. It’s all nice when people can agree on a single venue to send
their replies to, but that doesn’t happen every time.</p>
<p>Case in point, I’m getting <strong>5</strong> copies of a bunch of mails, through
the following <code>debian-*</code> lists: accessibility, boot, cd, devel,
project.</p>
<p>Needless to say: Reading, or marking a given mail as read once per
maildir rapidly becomes a burden.</p>
<h2>Solution</h2>
<p>I know some people use a duplicate killer at <code>procmail</code> time (hello
gregor) but I’d rather keep all mails in their relevant maildirs.</p>
<p>So here’s
<a href="http://mraw.org/blog/2014/08/11/mark-as-read-everywhere.pl">mark-read-everywhere.pl</a>
which seems to do the job just fine for my particular setup: all
maildirs below <code>~/mails/*</code> with the usual <code>cur</code>, <code>new</code>, <code>tmp</code>
subdirectories.</p>
<p>Basically, given a mail piped from <code>mutt</code>, compute a hash on various
headers, look at all new mails (<code>new</code> subdirectories), and mark the
matching ones as read (move to the nearby <code>cur</code> subdirectories, and
change suffix from <code>,</code> to <code>,S</code>).</p>
<p>Mutt key binding (where <em>X</em> is short for <em>cross</em> post):</p>
<pre><code>macro index X "<pipe-message>~/bin/mark-as-read-everywhere.pl<enter>"
</code></pre>
<p>This isn’t pretty or bulletproof but it already started saving time!</p>
<p>Now to wonder: <a href="http://xkcd.com/1205/">was it worth the time</a> to
<a href="http://xkcd.com/1319/">automate</a> that?</p>
Fun with ints, longs, and pointers!http://mraw.org/blog/2012/05/21/Fun_with_integers_longs_pointers/2013-12-17T02:11:29Z2012-05-20T22:20:00Z
<p>While investigating the case of some packages responsible for
uninstallability on the <code>s390x</code> architecture, I stumbled upon one
FTBFS on a single architecture, reported as
<a href="http://bugs.debian.org/673590">#673590</a> (<strong><insert some nice words
about software shipping with a decent testsuite></strong>).</p>
<p>Given the <code>int</code> versus <code>size_t</code> question was asked, I grabbed this old
(it’s UNIX!) reference:
<a href="http://www.unix.org/whitepapers/64bit.html">64-bit and Data Size Neutrality</a>.</p>
<p>Among other things, it describes the <em>“everything is 32-bit”</em> to <em>“64-bit
is becoming the new standard”</em> slow conversion process, where keeping
compatibility with existing applications was a high priority. It has a
nice description of so-called <em>C data models</em>, making it possible to
refer to them quickly: LP32 and ILP32 in the 32-bit world; ILP64,
LLP64, and LP64 in the 64-bit world. I won’t go into detail here, this
page has a nice table and lots of explanations about pros and cons for
those.</p>
<p>(On a personal note, I discovered those on <code>xorg-devel@</code> when I saw
patches floating around, which were about optimizing data sizes for
this or that data model, by picking the right types.)</p>
<p>While standards may be boring, this page makes it really easy to
understand which data types to use, to ensure the best 32/64-bit
compatibility possible. It’s even full of <strong>do</strong>s and <strong>don’t</strong>s. See the
second half (<strong>Porting Issues</strong>) for details.</p>
Thoughts on multiple webcamshttp://mraw.org/blog/2011/07/18/Thoughts_on_multiple_webcams/2013-12-17T02:11:29Z2011-07-17T23:00:00Z
<p>Keeping an eye on something with a webcam is really easy: just a
matter of setting up for example (c)vlc to multicast from
<code>/dev/videoX</code> to the network, and done.</p>
<p><strong>Example:</strong></p>
<pre><code># Streaming:
cvlc v4l2:///dev/video0 --no-audio --sout "#transcode{vcodec=mp2v,vb=3072}:std{access=udp,mux=ts,dst=224.0.0.1}"
# Playing, possibly on a remote machine:
vlc udp://@224.0.0.1:1234
</code></pre>
<p><strong>Notes:</strong></p>
<ul>
<li><code>v4l2</code> vs. <code>v4l</code> depending on the distribution (<code>sid</code> vs. <code>squeeze</code>).</li>
<li>Triple slash needed, two for the <code>v4l[2]://</code> part, one for the
device (<code>/dev/video0</code>).</li>
<li>Why this address? See
Wikipedia’s <a href="http://en.wikipedia.org/wiki/Multicast_address">Multicast address article</a>.</li>
</ul>
<p><strong>Possible problems:</strong></p>
<ul>
<li>If switches don’t have IGMP snooping enabled, throwing multicast at
them turns it into plain broadcast. In other words: instead of
having packets sent to the hosts subscribed to that multicast
address, everyone gets those packets, polluting the network.</li>
<li>Trying to stream from multiple devices may fail, due to USB
bandwidth issues. At least that can be seen with two random
devices using the <code>uvcvideo</code> module, getting <code>No space left on
device</code> when the second (c)vlc is started.</li>
</ul>
<p><strong>Where to go from here:</strong></p>
<ul>
<li><a href="http://www.firestorm.cx/fswebcam/">fswebcam</a> to the rescue! It can
capture a frame from a given device. The following options are
particularly useful: <code>--resolution</code> to request a specific
resolution (the device will fall back to another resolution if the
requested one is unavailable); <code>--frames</code> to specify the number of
frames to capture (lower noise, but possibly higher blurriness).</li>
<li>A tiny python wrapper around it makes it possible to set various
webcams, with different options (number of frames, delay between
two captures, resolution), operating at the same time thanks to the
help of the
<a href="http://docs.python.org/library/threading.html"><code>threading</code> module</a>. To
avoid encountering the same bandwidth issue, a tiny lock ensures a
single <code>fswebcam</code> instance runs at a time.</li>
<li>That also makes it possible to save all images as
<code>videoX-timestamp.jpg</code> and to “log”rotate them after a while, for
further viewing/reviewing during a given time window.</li>
<li><p>Thanks to <a href="http://www.imagemagick.org/">ImageMagick</a>, it’s also
trivial to process the captured images:</p>
<pre><code> # Only keep a rectangle of width W, height H, starting at the point (X,Y):
convert -crop WxH+X+Y capture.jpg capture.jpg
# Include a timestamp on the top-left corner:
convert -box white -font Fixed -draw "text 2,12 \"$timestamp\"" capture.jpg capture.jpg
</code></pre></li>
<li><p>Keeping a <code>videoX-current.jpg</code> symlink for each <code>/dev/videoX</code>
device is trivial.</p></li>
<li>Add to that a lightweight webserver and a single page showing the
most recent capture for each device thanks to the symlinks, with a
<a href="http://en.wikipedia.org/wiki/Meta_refresh">meta refresh</a> of say 5
seconds, and tada! Keeping an eye on multiple things at once. Of
course that’s not a real video, but that can fulfill some
needs.</li>
</ul>