XDXA http://xdxa.org/ en Tue, 04 Feb 2014 20:04:00 GMT http://xdxa.org/rss/ acrylamid 0.7.7 String Concatenation - Root of all Evil http://xdxa.org/2014/string-concatenation-root-of-all-evil/ <p>After college, I spent three years in software security acting as a <a href="https://en.wikipedia.org/wiki/Penetration_test">pen-tester</a>. It was my job to identify security vulnerabilities in the company's software via exploitation. During that time, string concatenation was the root cause of the vast majority of my findings. For that reason, it is my general philosophy that:</p> <blockquote> <p>If you're concatenating strings, you're doing it wrong.</p> </blockquote> <h3>What?</h3> <p>To make my point easier to grok, let's start with a simple example:</p> <div class="highlight"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">person</span><span class="o">.</span><span class="n">first_name</span> <span class="o">+</span> <span class="s2">&quot; &quot;</span> <span class="o">+</span> <span class="n">person</span><span class="o">.</span><span class="n">last_name</span> <span class="s1">&#39;Bob Smith&#39;</span> </pre></div> <p>We've joined a person's first and last name with a space. The concatenated value can be thought of as a space delimited string. In of itself, this is not an issue. But, what happens when we need to reverse this transformation? Easy enough. Using the previous output as our input, we split on the space character.</p> <div class="highlight"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">name</span> <span class="o">=</span> <span class="s2">&quot;Bob Smith&quot;</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot; &quot;</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">print</span> <span class="n">name</span> <span class="p">[</span><span class="s1">&#39;Bob&#39;</span><span class="p">,</span> <span class="s1">&#39;Smith&#39;</span><span class="p">]</span> </pre></div> <p>What happens if the first name contains a space? If an attacker controls that value, an injected space could break the application's intended behavior. It's conceivable to imagine that after <em>deserialization</em>, the value might be used in some security related decision.</p> <div class="highlight"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">name</span> <span class="o">=</span> <span class="s2">&quot;Bob Smith RealLastName&quot;</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot; &quot;</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">print</span> <span class="n">name</span> <span class="p">[</span><span class="s1">&#39;Bob&#39;</span><span class="p">,</span> <span class="s1">&#39;Smith&#39;</span><span class="p">,</span> <span class="s1">&#39;RealLastName&#39;</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">if</span> <span class="s2">&quot;Smith&quot;</span> <span class="o">==</span> <span class="n">name</span><span class="p">[</span><span class="mi">1</span><span class="p">]:</span> <span class="o">...</span> <span class="k">print</span> <span class="s2">&quot;Welcome Smith!&quot;</span> <span class="o">...</span> <span class="n">Welcome</span> <span class="n">Smith</span><span class="err">!</span> </pre></div> <p>To address the vulnerability, we have a few options:</p> <ul> <li>Validate the first name, rejecting requests with spaces</li> <li>Sanitize the first name, removing spaces</li> <li>Introduce an escape sequence and <em>escape</em> the names</li> </ul> <p>Validation and sanitization are appealing choices, but if the business logic forbids it, our only option is escaping. A simple solution would be to replace spaces with <code>'\ '</code> and backlashes with <code>'\\'</code>.</p> <div class="highlight"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">person</span><span class="o">.</span><span class="n">first_name</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">&#39; &#39;</span><span class="p">,</span> <span class="s1">&#39;\ &#39;</span><span class="p">)</span> <span class="o">+</span> <span class="s2">&quot; &quot;</span> <span class="o">+</span> <span class="n">person</span><span class="o">.</span><span class="n">last_name</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">&#39; &#39;</span><span class="p">,</span> <span class="s1">&#39;\ &#39;</span><span class="p">)</span> <span class="s1">&#39;Bob</span><span class="se">\\</span><span class="s1"> Smith RealLastName&#39;</span> </pre></div> <p>You might be inclined to argue that this is a sufficient fix (nevermind that it's not to spec). However, I'm here to suggest that's not true. Manually escaping and concatenating strings is implicitly brittle. Forgetting to apply escaping algorithm in one place is enough to introduce a severe vulnerability.</p> <p>The mistake here was <em>casually</em> performing serialization rather than utilizing some library, whose sole responsibility is serialization. This example is somewhat contrived, but the fundamental principal can be expanded to CSVs, XML, JSON, URIs, SQL, or any <em>formatted</em> data.</p> <h3>Examples</h3> <p>Let's break down a few exploits to demonstrate my point.</p> <h4>SQL</h4> <p>Here we have a simple <a href="https://en.wikipedia.org/wiki/SQL_injection">SQL injection</a> in PHP:</p> <div class="highlight"><pre><span></span><span class="cp">&lt;?</span> <span class="s2">&quot;SELECT * FROM table WHERE col=&#39;&quot;</span> <span class="o">.</span> <span class="nv">$input</span> <span class="o">.</span> <span class="s2">&quot;&#39;&quot;</span><span class="p">;</span> </pre></div> <p>If the input contains a single quote, <code>"'"</code>, the SQL query's structure is disrupted. An attacker can use this to alter the query. (This example is benign. There are significantly more malicious uses for SQL injection, but that's not what I'm here to discuss.)</p> <div class="highlight"><pre><span></span><span class="cp">&lt;?</span> <span class="nv">$input</span> <span class="o">=</span> <span class="s2">&quot;&#39; OR &#39;&#39;=&#39;&quot;</span><span class="p">;</span> <span class="nv">$query</span> <span class="o">=</span> <span class="s2">&quot;SELECT * FROM table WHERE col=&#39;&quot;</span> <span class="o">.</span> <span class="nv">$input</span> <span class="o">.</span> <span class="s2">&quot;&#39;&quot;</span><span class="p">;</span> <span class="k">echo</span> <span class="nv">$query</span><span class="p">;</span> <span class="nx">SELECT</span> <span class="o">*</span> <span class="nx">FROM</span> <span class="nx">table</span> <span class="nx">WHERE</span> <span class="nx">col</span><span class="o">=</span><span class="s1">&#39;&#39;</span> <span class="k">OR</span> <span class="s1">&#39;&#39;</span><span class="o">=</span><span class="s1">&#39;&#39;</span> </pre></div> <p>How do we fix it? There are three general approaches:</p> <div class="highlight"><pre><span></span><span class="cp">&lt;?</span> <span class="mf">1.</span> <span class="s2">&quot;...WHERE col=&#39;&quot;</span> <span class="o">.</span> <span class="nb">str_replace</span><span class="p">(</span><span class="s2">&quot;&#39;&quot;</span><span class="p">,</span> <span class="s2">&quot;&quot;</span><span class="p">,</span> <span class="nv">$x</span><span class="p">)</span> <span class="o">.</span> <span class="s2">&quot;&#39;&quot;</span> <span class="mf">2.</span> <span class="s2">&quot;...WHERE col=&#39;&quot;</span> <span class="o">.</span> <span class="nb">mysql_real_escape_string</span><span class="p">(</span><span class="nv">$x</span><span class="p">)</span> <span class="o">.</span> <span class="s2">&quot;&#39;&quot;</span> <span class="mf">3.</span> <span class="nv">$mysqli</span><span class="o">-&gt;</span><span class="na">prepare</span><span class="p">(</span><span class="s2">&quot;...WHERE col=?&quot;</span><span class="p">,</span> <span class="nv">$x</span><span class="p">)</span> </pre></div> <p>The first approach manually sanitizes the input by blacklisting the single quote character. This approach is problematic, because it doesn't account for all possible special characters. If the surrounding quotes were instead double quotes, this method would fail. When I've seen this type of sanitization, it is typically done at the beginning of the request, far away from the code which actually utilizes the variable. That increases the probability of such a mistake.</p> <p>The second approach utilizes a library call to escape the input. The last approach parameterizes the query, sending the query and its arguments separately to the database. Both the second and third approach utilize a library call, and both work for this query. If that's the case, what's wrong with using the version that utilizes string concatenation? What if the column was instead a number?</p> <div class="highlight"><pre><span></span><span class="cp">&lt;?</span> <span class="s2">&quot;...WHERE number=&quot;</span> <span class="o">.</span> <span class="nb">mysql_real_escape_string</span><span class="p">(</span><span class="nv">$x</span><span class="p">);</span> </pre></div> <p>There are no characters to escape in this case! An injection like <code>"0 OR 1=1"</code> will still work. If the variable <code>$x</code> had been validated as an integer prior to use, there wouldn't have been an issue. However that's a brittle solution, because a simple mistake (or refactor) could still compromise the application's security.</p> <p>What's really wrong here? The query <em>is</em> the formatted data. The library call is used to escape a value which is being <em>serialized</em> into the query. That's the responsibility of the library. By using string concatenation, we've effectively written part of a SQL serializer.</p> <p>(Learn more about <a href="http://www.codinghorror.com/blog/2005/04/give-me-parameterized-sql-or-give-me-death.html">motivations to use parameterized queries</a>.)</p> <h4>URI</h4> <p>The URI is a fascinating point to perform injection attacks upon. They are extremely easy to construct without a library, which means it's done frequently, and the results work <em>most</em> of the time. As more applications move towards <a href="https://en.wikipedia.org/wiki/Service-oriented_architecture">service oriented architectures</a>, URIs are increasingly used as a component of the communication. If unescaped input is placed into the URI, it might be possible to alter the application's intended behavior.</p> <p>In the following example, we have a web application which transfers funds by dispatching to an HTTP service. The user controls the variable <em>amount</em>, but not <em>user</em> or <em>target</em>. The service domain <em>s.xdxa.org</em> is not internet accessible. We can assume the backend service will validate <em>amount</em> as a positive integer prior to use.</p> <div class="highlight"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">print</span> <span class="s1">&#39;http://s.xdxa.org/pay?amount=&#39;</span> <span class="o">+</span> <span class="n">amount</span> <span class="o">+</span> <span class="s1">&#39;&amp;from=&#39;</span> <span class="o">+</span> <span class="n">user</span> <span class="o">+</span> <span class="s1">&#39;&amp;to=&#39;</span> <span class="o">+</span> <span class="n">target</span> <span class="s1">&#39;http://s.xdxa.org/pay?amount=100&amp;from=eve&amp;to=bob&#39;</span> </pre></div> <p>To force Bob to pay Eve, we only need to alter the URI structure.</p> <div class="highlight"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">amount</span> <span class="o">=</span> <span class="s1">&#39;100&amp;from=bob&amp;to=eve#&#39;</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">print</span> <span class="o">...</span> <span class="s1">&#39;.../pay?amount=100&amp;from=bob&amp;to=eve#&amp;from=eve&amp;to=bob&#39;</span> </pre></div> <p>The fragment (everything after the <code>'#'</code>) is dropped before making the backend request. As a result, the service's log will look completely normal. To fix this, use a library to generate URIs. This is typically accomplished by using either a builder or providing a map of "parameter to argument"s.</p> <p>Unfortunately, libraries for RESTful URIs, where arguments may be part of the path, are more scarce. In that circumstance, concatenation + escaping may be your only option. However, if you do that, segment that code into its own domain, i.e., create a library. The Java library <a href="https://github.com/damnhandy/Handy-URI-Templates">Handy URI Templates</a> is a good model for generating said URIs.</p> <p>Additionally, be careful in other sections of the URI. Other components of the URI have different rules, i.e., scheme and authority. If you want to know more, <a href="http://lcamtuf.coredump.cx/tangled/">The Tangled Web</a> has an entire chapter dedicated to the URL.</p> <h4>HTML</h4> <p><a href="https://en.wikipedia.org/wiki/Cross-site_scripting">XSS</a> (Cross-Site Scripting) is a vulnerability where an attacker is able to inject JavaScript into a page, which is then executed in a victim's browser. It's a subset of the attacks against poorly serialized HTML. The attack is made possible by the fact that HTML mingles mark-up and code. In fact, HTML is composed of a number of different data formats within a single document:</p> <ul> <li>HTML Tags</li> <li>JavaScript</li> <li>CSS</li> <li>URIs</li> </ul> <p>When represented in a object structure, we refer to the page as a DOM (document object model). However to get that page to the browser, it's serialized as a string and transmitted over HTTP. It is in that process that XSS is made possible. To make this concrete, let's take a look at an example. What happens if an attacker includes a script tag in the URL <code>arg</code> parameter? (Again, this is a simplistic example. There are significantly more malicious XSS attacks.)</p> <div class="highlight"><pre><span></span><span class="x">&lt;span&gt;</span><span class="cp">&lt;?php</span> <span class="k">echo</span> <span class="nv">$_GET</span><span class="p">[</span><span class="s1">&#39;arg&#39;</span><span class="p">];</span> <span class="cp">?&gt;</span><span class="x">&lt;/span&gt;</span> <span class="x">becomes</span> <span class="x">&lt;span&gt;Hello! &lt;script&gt;alert(1);&lt;/script&gt;&lt;/span&gt;</span> </pre></div> <p>The user content altered the structure of the document. To fix this, we could escape the variable prior to output. However, that is simply string concatenation. The output is exactly the same.</p> <div class="highlight"><pre><span></span><span class="x">&lt;span&gt;</span><span class="cp">&lt;?php</span> <span class="k">echo</span> <span class="nb">htmlentities</span><span class="p">(</span><span class="nv">$_GET</span><span class="p">[</span><span class="s1">&#39;arg&#39;</span><span class="p">]);</span> <span class="cp">?&gt;</span><span class="x">&lt;/span&gt;</span> <span class="x">is the same as</span> <span class="cp">&lt;?</span> <span class="k">echo</span> <span class="s1">&#39;&lt;span&gt;&#39;</span> <span class="o">.</span> <span class="nb">htmlentities</span><span class="p">(</span><span class="nv">$_GET</span><span class="p">[</span><span class="s1">&#39;arg&#39;</span><span class="p">])</span> <span class="o">.</span> <span class="s1">&#39;&lt;/span&gt;&#39;</span><span class="p">;</span> </pre></div> <p>This means, it falls prey to all the same risks as above. One missed escape, or escaping for the wrong context, and we have a vulnerability.</p> <div class="highlight"><pre><span></span><span class="x">&lt;span href=&quot;</span><span class="cp">&lt;?php</span> <span class="k">echo</span> <span class="nb">htmlentities</span><span class="p">(</span><span class="nv">$_GET</span><span class="p">[</span><span class="s1">&#39;arg&#39;</span><span class="p">]);</span> <span class="cp">?&gt;</span><span class="x">&quot;&gt;...&lt;/span&gt;</span> <span class="x">becomes</span> <span class="x">&lt;span href=&quot;javascript:alert(1);&quot;&gt;...&lt;/span&gt;</span> </pre></div> <p>But wait! XSS is <em>really</em> because a programmer failed to check for JavaScript within the user inlined content. That's true, in the implementation of HTML we know and love. If the issue wasn't truly with the failure to preserve the document structure, we could simply introduce a tag which would disable all scripts within the child element. That hasn't happened, because the issue is really about the injection. (As an aside, avoid using HTML for user supplied markup. It's a losing battle.)</p> <p>So what do I recommend? Templating engines. (Keep in mind, templating engines may <em>not</em> automatically escape variables within your output. Escaping must be configured on a per-engine basis.)</p> <div class="highlight"><pre><span></span><span class="p">&lt;</span><span class="nt">span</span><span class="p">&gt;</span>{{ input.arg }}<span class="p">&lt;/</span><span class="nt">span</span><span class="p">&gt;</span> </pre></div> <p>That having been said, automatic escaping is typically <em>not</em> a panacea. Why? The escaping is normally context unaware, which means it's performed for HTML tags. If your templating engine allows bad markup, it's doubtful that it understands the context for which it's escaping. Where can this bite you?</p> <ol> <li>Tag Attributes</li> <li>URIs</li> <li>CSS</li> <li>JavaScript <br /> (It is rarely safe to output content into a script tag. Avoid this whenever possible.)</li> </ol> <p>Another even safer approach is to construct your page as a DOM. This will ensure tags, attributes, and possible URIs are properly escaped, but probably not CSS or Javascript. Almost no frameworks actually do this, because it's cumbersome and (probably) more cpu/memory costly.</p> <p>Tangential from string concatenation, is a perfect templating engine enough? Unfortunately, browsers suck (from a security perspective). You could trace this back to the browser wars, where browsers which refused to render bad markup were labeled at fault. Regardless of the history, browsers have extremely "flexible" parsers. As a result, even if a document looks rightish, it might still contain an XSS. While I generally do not recommend sanitizing user input (it feels too much like a blacklist), input destined for HTML is an exception for the reasons outlined above. See <a href="https://www.owasp.org/index.php/XSS_%28Cross_Site_Scripting%29_Prevention_Cheat_Sheet#RULE_.236_-_Sanitize_HTML_Markup_with_a_Library_Designed_for_the_Job">OWASP's Sanitization Recommendations</a> for candidate libraries.</p> <p>You could easily write a book on injection attacks against HTML. For that reason, I'm going to stop here. Want to know more?</p> <ul> <li><a href="http://html5sec.org/">HTML5 Security Cheatsheet</a></li> <li><a href="https://www.owasp.org/index.php/XSS_Filter_Evasion_Cheat_Sheet">XSS Filter Evasion Cheat Sheet</a></li> <li><a href="http://lcamtuf.coredump.cx/postxss/">Postcards from the post-XSS world</a></li> <li><a href="http://www.slideshare.net/x00mario/the-innerhtml-apocalypse">The innerHTML Apocalypse</a></li> <li><a href="http://lcamtuf.blogspot.co.uk/2010/10/http-cookies-or-how-not-to-design.html">HTTP cookies, or how not to design protocols</a></li> <li><a href="http://lists.webappsec.org/pipermail/websecurity_lists.webappsec.org/2006-March/000843.html">Path Insecurity</a></li> <li><a href="http://www.xdxa.org/presentations/xss-csrf/">My XSS/CSRF Slide Deck</a></li> </ul> <h4>JSON</h4> <p>To avoid making this article much longer, I'm not going to break out a JSON example. However, I do want to mention it, because I have frequently seen engineers construct JSON from a string. Don't do this.</p> <h3>Embedded Variables</h3> <p>Note, there is no distinction between embedding a variable and string concatenation. Neither of the following two examples address the serialization problem:</p> <div class="highlight"><pre><span></span><span class="cp">&lt;?</span> <span class="s2">&quot;</span><span class="si">$first</span><span class="s2"> </span><span class="si">$last</span><span class="s2">&quot;</span> <span class="o">===</span> <span class="nv">$first</span> <span class="o">.</span> <span class="s1">&#39; &#39;</span> <span class="o">.</span> <span class="nv">$last</span><span class="p">;</span> </pre></div> <p>Or with sprintf like substitution:</p> <div class="highlight"><pre><span></span><span class="s2">&quot;</span><span class="si">%s</span><span class="s2"> </span><span class="si">%s</span><span class="s2">&quot;</span> <span class="o">%</span> <span class="p">(</span><span class="n">person</span><span class="o">.</span><span class="n">first_name</span><span class="p">,</span> <span class="n">person</span><span class="o">.</span><span class="n">last_name</span><span class="p">)</span> </pre></div> <h3>Logging</h3> <p>Before we conclude, I want to make a special note with regards to logging. In my code, casual string concatenation most frequently occurs in logging statements. Logs are typically loosely formatted, newline delimited files. Loosely structured output is <em>mostly</em> fine when intended for human consumption. However, this casual serialization format is a nightmare for log parsers.</p> <p>For this reason, one must be careful when logging output which will eventually be consumed by some downstream process. While not strictly the responsibly of the logger, HTML log viewers have been susceptible to XSS attacks <a href="http://www.cvedetails.com/cve/CVE-2012-2094">in the past</a>.</p> <p>See <a href="http://cwe.mitre.org/data/definitions/117.html">CWE-117</a> and <a href="http://capec.mitre.org/data/definitions/106.html">CAPEC-106</a> for more information.</p> <h3>Clarification</h3> <p>This concept is not new or innovative. The weakness I'm describing is actually <a href="http://cwe.mitre.org/data/definitions/707.html">CWE-707: Improper Enforcement of Message or Data Structure</a>, and it is not exclusive to string concatenation or even strings. I wag my finger at string concatenation, because it makes these mistakes easy. And consequentially, common.</p> <h3>Hyperboles</h3> <p>The title is mostly meant as tongue and cheek, but I'm quite serious in the severity of this topic. In my opinion there are only two places string concatenation should occur:</p> <ol> <li>Serialization/Deserialization Libraries</li> <li>Unformatted Data</li> </ol> <p>Many security issues simply do not have an easy solution, e.g. <a href="https://en.wikipedia.org/wiki/Side_channel_attack">side channel attacks</a>. However, if engineers consistently use libraries for serialization, the majority of injection based attacks will simply go away.</p> <p>Is it possible to write secure code with concatenation? Yes. However, this code will almost always be brittle. You might understand the security considerations, but the next engineer probably will not. Writing secure code isn't just preventing a certain type of attack. It's following best practice throughout the entire code base to set the standard which avoids careless mistakes.</p> Tue, 04 Feb 2014 20:04:00 GMT tag:xdxa.org,2014-02-04:/2014/string-concatenation-root-of-all-evil Vertical Rhythm http://xdxa.org/2013/vertical-rhythm/ <p>I've always found interface usability fascinating. Subtle differences in coloring, spacing, and layout can drastically impact the functionality of your application. I think it's important for programmers to understand some of the basic principals of usability.</p> <p>A few years back, I read about a design concept referred to as Vertical Rhythm. The way I like to think about it is to imagine a web page as a sheet of lined paper. All text is written upon the lines, which is referred to as the baseline. A heading could pass through multiple lines, but it shouldn't impact the following text's ability to align with the baseline.</p> <p>Why go through the effort of creating a repetitive baseline? People are extremely good at pattern recognition. If we utilize this skill set, our interfaces will feel natural to use. When skimming text, a consistent baseline makes it easier to read, because the text lands in the location expected by our brains.</p> <p>I set out today with the intent to write a script to assist verification of a page's baseline, but it seems someone has already accomplished this. <a href="http://keyes.ie/things/baseliner/">Baseliner</a> provides a bookmarklet. When executed on a page, it will overlay a series of horizontal lines. These lines can be adjusted to match your page's line-height and layout. At the time of writing, this page has a grid size of 21 with an offset of 15.</p> <p>Like all things CSS, numerous styling issues can arise. When styling your web application, keep the following in mind:</p> <ul> <li>Top margins will collapse into previous tag's bottom margin.</li> <li>Images may not be a multiple of the line-height.</li> <li>Headings whose line-height is not a multiple of the baseline line-height will cause issues if wrapped.</li> </ul> <p>When I initially discovered vertical rhythm, there wasn't much written on the web covering the subject. That has changed. If this interests you, I suggest check out the following:</p> <ul> <li><a href="http://coding.smashingmagazine.com/2012/12/17/css-baseline-the-good-the-bad-and-the-ugly/">CSS Baseline: The Good, The Bad and The Ugly</a></li> <li><a href="http://www.creativebloq.com/maintain-vertical-rhythm-css-and-jquery-2124274">Maintaining Vertical Rhythm</a></li> </ul> <p>How important is this really? One of my favorite browser extensions, <a href="http://readability.com/">Readability</a>, which reformats poorly laid out blogs, doesn't maintain vertical rhythm. Nor does Wikipedia. Ironically, neither does the Baseliner page or the two links above. In fact, I couldn't find a single site that followed this advice.</p> <p>Does that mean it's wrong? The lack of a consistent baseline on the web could be due to the difficulties in establishing rhythm with CSS. Or, it could be that there are more optimal spacing practices. In the end, I don't really know whether it's right or wrong. Regardless, this principle isn't dogma. It's a tool, just like the color wheel. For someone who is not a designer, I find it useful in establishing an application's foundation. Next time you're working on a layout, give it a try and see what you think.</p> Sun, 17 Nov 2013 17:54:00 GMT tag:xdxa.org,2013-11-17:/2013/vertical-rhythm Vagrant http://xdxa.org/2013/vagrant/ <p>I love the notion of a clean, repeatable install. In my side projects, I've historically accomplished this through strict documentation practices and directory organization. However, this process inevitably breaks down for me when a machine becomes multi-purpose.</p> <p>Either locally or in the cloud, I'll end up installing packages, hacking together configuration, or otherwise making the environment murky. In fact, the server that currently hosts this blog is a perfect example. This instance hosts this site, riiga.net, and has been a sandbox for toy projects I've worked on over the past two years.</p> <p>While I'm reasonably confident I can redeploy this machine within a couple of hours, it would be far more flexible if I could do this in minutes. Or even better, selectively re-deploy projects to alternate hosts.</p> <p>For this reason, <a href="http://www.vagrantup.com/">Vagrant</a> has become my new favorite toy. Vagrant provides simple VM construction and provisioning. This enables easy provisioning of development environments, testing large system upgrades, and migrating projects to alternative cloud providers.</p> <p>To begin exploring the world of automated provisioning, I decided to create a VM that would serve the <a href="http://docs.services.mozilla.com/howtos/run-sync.html">Mozilla Sync server</a>. I've hosted the <a href="https://github.com/bheiskell/vagrant-mozilla-sync">Vagrant Mozilla Sync Configuration</a> on my GitHub. (Disclaimer: this was a warm-up project. All dependencies are automatically pulled from other source controls. Meaning, this will likely break in the future.)</p> <p>Isn't this overkill? It certainly seemed so... that is, until the first VM corrupted. When that happened, I simply re-provisioned a new instance. About an hour later, I was back up and running with minimal engineering effort.</p> <p>What next? I've been itching to try Linux Mint. And what better way than to automatically provision my standard desktop deployment!</p> Sun, 28 Apr 2013 11:21:00 GMT tag:xdxa.org,2013-04-28:/2013/vagrant Acrylamid http://xdxa.org/2013/acrylamid/ <p>For a long while I've wanted to host this blog rather than utilize a service like Google's blogger. My biggest hesitation was blogging software tends to be a common means of getting pwned. I could write straight HTML, but then the content is mingled with the markup - it feels dirty. FrontPage you say? Well, that's just not cool enough.</p> <p>Enter <a href="https://github.com/posativ/acrylamid">Acrylamid</a>, a simple, but powerful means of generating a static website. Out of the box, it supports:</p> <ul> <li><a href="http://daringfireball.net/projects/markdown/">Markdown</a></li> <li><a href="http://jinja.pocoo.org/docs/">Jinja2</a> templating</li> <li>Code highlighting with <a href="http://pygments.org/">Pygments</a></li> </ul> <p>To convert the contents of the old blog, I downloaded it using <a href="https://en.wikipedia.org/wiki/Google_Takeout">Google Takeout</a>. Through a combination of xpath, <a href="https://github.com/aaronsw/html2text">html2text</a>, and some python hackery, I converted the previous entries to markdown.</p> <p>For styling, I'm utilized the <a href="http://www.initializr.com/">responsive template from Initializr</a>. While I've used both Bootstrap and their HTML5 reset in the past, this is the first time I've gone responsive. It's surprisingly straight forward and seems to work well on my android devices.</p> <p>But wait! Where are the comments? Sorry folks - no comments. I debated about using a service like <a href="http://disqus.com/">Disqus</a>, but I just dislike the idea of including a third-party service (and consequentially tracking) on my site. If you have feedback, see the about page. There are numerous means of communicating with me there.</p> Sat, 13 Apr 2013 19:09:00 GMT tag:xdxa.org,2013-04-13:/2013/acrylamid CakePHP's error handling http://xdxa.org/2011/cakephp-s-error-handling/ <p>Whenever possible, I like to trigger the appropriate HTTP status code for an error. CakePHP has a mechanism to throw 404 errors by simply calling:</p> <div class="highlight"><pre><span></span><span class="cp">&lt;?php</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="na">cakeError</span><span class="p">(</span><span class="s1">&#39;error404&#39;</span><span class="p">);</span> </pre></div> <p>This is handy when a request to a view action passes an id that doesn't actuallyexist. However, what about 403 or 500 errors? If you're like me, you probably stumbled upon <a href="http://book.cakephp.org/view/154/Error-Handling">AppError in the CakePHP book</a>.</p> <p>I defined a simple error403 method, and used it where appropriate throughout my application. But when I rolled out to production, every 403 became a 404! Upon a bit of digging around in the code, I found the root of the problem in the ErrorHandler constructor. In version 1.2.9, you can find this in cake/libs/error.php:</p> <div class="highlight"><pre><span></span><span class="cp">&lt;?php</span> <span class="k">if</span> <span class="p">(</span><span class="nv">$method</span> <span class="o">!==</span> <span class="s1">&#39;error&#39;</span><span class="p">)</span> <span class="p">{</span> <span class="k">if</span> <span class="p">(</span><span class="nx">Configure</span><span class="o">::</span><span class="na">read</span><span class="p">()</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span> <span class="nv">$method</span> <span class="o">=</span> <span class="s1">&#39;error404&#39;</span><span class="p">;</span> <span class="k">if</span> <span class="p">(</span><span class="nb">isset</span><span class="p">(</span><span class="nv">$code</span><span class="p">)</span> <span class="o">&amp;&amp;</span> <span class="nv">$code</span> <span class="o">==</span> <span class="mi">500</span><span class="p">)</span> <span class="p">{</span> <span class="nv">$method</span> <span class="o">=</span> <span class="s1">&#39;error500&#39;</span><span class="p">;</span> <span class="p">}</span> <span class="p">}</span> <span class="p">}</span> </pre></div> <p>(Side note: Although in production error500 technically can be called, $code doesn't appear to be declared in the constructors scope. Also, the method error500 isn't actually defined, which adds to my confusion.)</p> <p>In production, all methods translate to either 404 or 500 errors! Now let me note, this does not appear to be mentioned anywhere in the book... So, how do we resolve this? Well, I found <a href="http://nuts-and-bolts-of-cakephp.com/2008/08/29/dealing-with-errors-in-cakephp/">someone having a similar problem</a>.</p> <p>Teknoid sets debug to 1 when the error handler constructor is called. This causes the error handler to believe it's in debug mode and run whatever error method is passed its way.</p> <p>But, there's a very important distinction. Teknoid overrides his _outputMessage method to send an email to himself, not to display errors to users. If you were to simply set debug to 1 in the constructor, all CakePHP errors would be displayed to the user. This is why only error404 is allowed in vanilla CakePHP.</p> <p>To resolve this, I created a white-list of allowed error handlers. When these error handlers are requested, I enable debug mode.</p> <div class="highlight"><pre><span></span><span class="cp">&lt;?php</span> <span class="k">class</span> <span class="nc">AppError</span> <span class="k">extends</span> <span class="nx">ErrorHandler</span> <span class="p">{</span> <span class="k">var</span> <span class="nv">$allowedMethods</span> <span class="o">=</span> <span class="k">array</span><span class="p">(</span> <span class="s1">&#39;error403&#39;</span><span class="p">,</span> <span class="s1">&#39;error500&#39;</span><span class="p">,</span> <span class="p">);</span> <span class="k">function</span> <span class="nf">__construct</span><span class="p">(</span><span class="nv">$method</span><span class="p">,</span> <span class="nv">$messages</span><span class="p">)</span> <span class="p">{</span> <span class="k">if</span> <span class="p">(</span><span class="nb">in_array</span><span class="p">(</span><span class="nv">$method</span><span class="p">,</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="na">allowedMethods</span><span class="p">))</span> <span class="p">{</span> <span class="nx">Configure</span><span class="o">::</span><span class="na">write</span><span class="p">(</span><span class="s1">&#39;debug&#39;</span><span class="p">,</span> <span class="mi">1</span><span class="p">);</span> <span class="p">}</span> <span class="k">parent</span><span class="o">::</span><span class="na">__construct</span><span class="p">(</span><span class="nv">$method</span><span class="p">,</span> <span class="nv">$messages</span><span class="p">);</span> <span class="p">}</span> <span class="k">function</span> <span class="nf">error403</span><span class="p">()</span> <span class="p">{</span> <span class="c1">// I do some logging here to audit</span> <span class="c1">// who made the forbidden request</span> <span class="nb">header</span><span class="p">(</span><span class="s2">&quot;HTTP/1.1 403 Forbidden&quot;</span><span class="p">);</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="na">controller</span><span class="o">-&gt;</span><span class="na">set</span><span class="p">(</span><span class="k">array</span><span class="p">(</span> <span class="s1">&#39;name&#39;</span> <span class="o">=&gt;</span> <span class="nx">__</span><span class="p">(</span><span class="s1">&#39;403 Forbidden&#39;</span><span class="p">,</span> <span class="k">true</span><span class="p">),</span> <span class="s1">&#39;title&#39;</span> <span class="o">=&gt;</span> <span class="nx">__</span><span class="p">(</span><span class="s1">&#39;403 Forbidden&#39;</span><span class="p">,</span> <span class="k">true</span><span class="p">),</span> <span class="p">));</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="na">_outputMessage</span><span class="p">(</span><span class="s1">&#39;error403&#39;</span><span class="p">);</span> <span class="p">}</span> <span class="o">...</span> <span class="nx">error500</span> <span class="k">and</span> <span class="nx">so</span> <span class="nx">on</span> <span class="o">...</span> <span class="p">}</span> </pre></div> <p>This is less than ideal, because turning on debug may have other implications throughout your application. We could reimplement (or copy and paste) the constructor with allowedMethods in mind, but that feels even more hack to me.</p> Thu, 17 Feb 2011 09:04:00 GMT tag:xdxa.org,2011-02-17:/2011/cakephp-s-error-handling Asus P4C800 Rev 2 http://xdxa.org/2009/asus-p4c800-rev-2/ <p>This board has been my latest personal hell. Back in the year 2005, my Asus P4C800 Deluxe fried. After taking their time, Asus sent me a replacement board. I simply assumed that it was a Deluxe and reinstalled it. Unfortunately, it was actually a P4C800 Rev 2. There are three models of the P4C800: P4C800, P4C800 deluxe, and P4C800-e deluxe. From what I can tell online, the Rev 2 is actually the e deluxe, but it looks like a hacked together mix between the P4C800 and the e deluxe.</p> <p>This is where I made a mistake. One night when trying to get suspend to work in Linux, I decided to flash this board. I flashed it without checking what bios version was already on there. Stupid. Worse, since I knew it wasn't the deluxe (because the updater rejected the bios), I assumed it was the e deluxe. As would be expected, after the flash, the system wont boot. Flash forward a few days, and I've managed to force a flash downgrading the system to a P4C800.</p> <p>Now looking back, I honestly can't remember what the board was originally and which bios I first forced onto the board. Never flash bios when tired and in a rush. I still can't believe I did that. Anyways, the system suffered from random shutoffs. I assumed this was due to the flash. (Honestly I can't be sure one way or the other now.) Over time these flickers grew increasingly annoying. So, I tried another flash. That time I managed to brick the board.</p> <p>Yesterday I received two replacement bios chips. (One is a backup for hot swapping.) The latest version of the P4C800-e deluxe bios works, but it wont reboot. The latest version of the P4C800 bios works and will reboot. Both suffer from the intermittent shutoffs, and both have bios errors referring to the IRQ routing table. At this stage, I'm pretty sure the problem either isn't a bios issue or at least isn't a bios issue that I can resolve.</p> <p>I installed this board into a new case with new hardware and the shutoff problems ensued. This has brought me to the conclusion that the problem has to be with the board. Removing all hardware except the bare bones still results in the shutoff. Another irritation is the problem wont normally occur when the system is idle. Also odd is the problems happen most in XP, less in Ubuntu, and the least in Windows 7. I don't know what that means exactly, but it suggests that whatever is triggering the problem may be a deprecated technique that modern OS's don't use. Though it may be that the board is just getting old and needs to be tossed.</p> <p>However, the two things that keep bring me back to suspecting the bios are: the coinciding timing of the first shut offs / bios flash and the fact that the board looks like a P4C800 with a Rev 2 sticker slapped on it instead of a P4C800-e deluxe board. There might be some bios out there that will stabilize this system that I have yet to find. I should probably contact Asus.</p> <p>Although I'm sure this isn't the conclusion to this story, it should be noted that suspend still doesn't work. :(</p> <p>Edit - 28/10/2010: This is the conclusion, because the pieces are now sitting in a box. My brother upgraded and let me keep his old hardware, which is newer than the P4C800. I'm now looking for a good way to get rid of this old hardware.</p> Wed, 18 Nov 2009 09:49:00 GMT tag:xdxa.org,2009-11-18:/2009/asus-p4c800-rev-2 GBA Development on a Mac http://xdxa.org/2009/gba-development-on-a-mac/ <p>In my media devices architecture class, we are developing on the GBA. Unfortunately, most of the students are developing on windows machines, which leaves me and a few others the odd men out. On macs, we're using devkitarm as our compiler with a provided makefile. Unfortunately, the Makefile had a few flags misconfigured.</p> <p>The stock makefile worked perfectly until we started dealing with sound. The problem was two fold:</p> <ol> <li>Setting the interrupt handler would cause the GBA to reset</li> <li>Sound files larger than thirty seconds would overflow the ewram</li> </ol> <p>There were two macros in the make file that were used for compiling and linking respectively:</p> <div class="highlight"><pre><span></span><span class="nv">MODEL</span> <span class="o">=</span> -mthumb -mthumb-interwork <span class="nv">SPECS</span> <span class="o">=</span> -specs<span class="o">=</span>gba_mb.specs </pre></div> <p>The model affects how the assembly is compiled, and the specs defines the way the linker will link the c objects together. I figured out that mthumb wasn't needed by looking at the windows makefile. Removing it resolved the rebooting issue.</p> <p>I knew that the overflow had to do with the linker, because that's where it failed. I was at the point where I was actually trying to modify the linker script to manually force the audio memory to go to the right region when I realized there was a second gba .specs file. Switching to that file resolved the issue.</p> <p>The resulting makefile macros are as followed:</p> <div class="highlight"><pre><span></span><span class="nv">MODEL</span> <span class="o">=</span> -mthumb-interwork <span class="nv">SPECS</span> <span class="o">=</span> -specs<span class="o">=</span>gba.specs </pre></div> <p>I am not a compiler expert. So, I can't tell you the full implications of changing these flags, but it did work for me. I'm still working on what I believe to be a compiler bug and plan to submit a bug report soon. If it turns out to be interesting, I'll make another post here.</p> Fri, 30 Oct 2009 08:19:00 GMT tag:xdxa.org,2009-10-30:/2009/gba-development-on-a-mac Microsoft Natural Ergonomic http://xdxa.org/2009/microsoft-natural-ergonomic/ <p>I decided I wanted to enable the zoom key on my "Microsoft Natural Ergonomic Keyboard 4000" under Linux. Ideally, this would be used for scrolling up and down a page instead of triggering Compiz's zoom feature.</p> <p>Unfortunately, the kernel fix is triaged on <a href="https://bugs.launchpad.net/ubuntu/+source/linux/+bug/264287">ubuntu's bugtracker</a>. This of course meant recompiling the kernel manually (or semi-manually since Ubuntu makes this task relatively easy). You'll notice, if you care to check, that there is a patch in the bugtracker to remap the zoom and spellcheck keys to something that xev (and therefore X11) can read.</p> <p>If you don't know how to recompile your kernel, this fix probably isn't for you. However, should you choose to proceed anyways (or if you're just not familiar with recompiling your kernel under Ubuntu), you can find <a href="https://help.ubuntu.com/community/Kernel/Compile#AltBuildMethod">instructions for recompiling Ubuntu's kernel here</a>.</p> <p>Once you've downloaded and uncompressed the source, edit "/usr/src/linux- source-*/include/linux/input.h". Search for ZOOMIN, ZOOMOUT, and SPELLCHECK and change these constants to match the following:</p> <div class="highlight"><pre><span></span><span class="cp">#define KEY_ZOOMIN 246 </span><span class="cm">/* AC Zoom In */</span><span class="cp"></span> <span class="cp">#define KEY_ZOOMOUT 247 </span><span class="cm">/* AC Zoom Out */</span><span class="cp"></span> <span class="p">...</span> <span class="cp">#define KEY_SPELLCHECK 235 </span><span class="cm">/* AL Spell Check */</span><span class="cp"></span> </pre></div> <p>Keep in mind that if some other input device already uses these key codes, then you'll have a conflict. Otherwise, once you've recompiled your kernel, you should be able to access these buttons in Gnome now.</p> <p><strong>Additional Information</strong></p> <p>What does this do and why do we do it? From my understanding, the reason this is a problem is X11 can't read key codes above a 255. Although this is a bug with X11, we can modify the kernel to remap these keys to a value lower. This enables X11 to read them and thus Gnome to use them.</p> <p>I dislike patch files for problems that are not going to be fixed, because they often break on the next kernel release. This is why I chose to manually edit the file instead of offer a patch file</p> <p>If you recompile your kernel, you'll likely have to recompile any other third party drivers you have installed. For instance, I had to recompile my Nvidia driver after rebooting. I also had to purge the nvidia-common package.</p> <p><strong>Mapping to Page Up / Down</strong></p> <p>Next, I wanted to map 246 and 247 to page up and page down respectively. To do this, I edited my ~/.Xmodmap file to include:</p> <div class="highlight"><pre><span></span>keycode 254 = Prior keycode 255 = Next </pre></div> <p>You can activate this by running "xmodmap ~/.Xmodmap" or adding the command to startup applications. However, the next time you log in, I think you'll get an option to enable this automatically without having to add it to startup apps. (I haven't setup xmodmap in a while, because I already use it to swap caplocks and escape... yes, I'm a vim user.)</p> <p>Edit: I should note that I have since reverted to a stock kernel. For some reason the recompiled kernel would kill my mouse periodically.</p> Mon, 26 Oct 2009 17:09:00 GMT tag:xdxa.org,2009-10-26:/2009/microsoft-natural-ergonomic Mac OS X Snow Leopard Port Wine http://xdxa.org/2009/mac-os-x-snow-leopard-port-wine/ <p>Currently, wine is not happy with my 64bit upgrade to Snow Leopard. Just for the record, I find no fault in either port or wine for this, because I was an early adopter. I upgraded port with a simple reinstall. (I know there are more elegant solutions than this, but I was being lazy.) After upgrading port, installing wine resulted in the following error:</p> <div class="highlight"><pre><span></span>Error: wine 1.0.1 is not compatible with Mac OS X 10.6 or later. Until wine 1.2 is released, please install wine-devel instead. </pre></div> <p>This is pretty straightforward. Obviously, the next step was "port install wine-devel". However, I was promptly stopped with another error in the following format:</p> <div class="highlight"><pre><span></span>Error: You cannot install wine-devel <span class="k">for</span> the architecture<span class="o">(</span>s<span class="o">)</span> i386 Error: because /opt/local/lib/libjpeg.dylib only contains the architecture<span class="o">(</span>s<span class="o">)</span> x86_64. Error: Try reinstalling the port that provides /opt/local/lib/libjpeg.dylib with the +universal variant. Error: Target org.macports.extract returned: incompatible architectures in dependencies </pre></div> <p>I'm not too familiar with port or the complexities of 64bit architectures working with 32bit binaries, but I do understand the principle behind OS X's universal (fat) binaries. After googling around a bit I discovered that the way to compile in the universal format is with the following command:</p> <div class="highlight"><pre><span></span>sudo port upgrade --enforce-variants jpeg +universal </pre></div> <p>The part that hung me up was figuring out which package was associated with that file. After digging around in the man page I found the following solution:</p> <div class="highlight"><pre><span></span>port provides /opt/local/lib/libjpeg.dylib </pre></div> <p>That's it! Just keep installing the dependencies and retrying "sudo port install wine-devel". After about eight dependencies are recompiled, you'll be finished. Good luck.</p> Wed, 21 Oct 2009 04:56:00 GMT tag:xdxa.org,2009-10-21:/2009/mac-os-x-snow-leopard-port-wine