Stop Docker Container properly when running systemd

You need to use docker run --stop-signal=SIGKILL ...  to stop a docker container properly, when the container is running systemd inside.

Reference: https://bugzilla.redhat.com/show_bug.cgi?id=1201657

Postfix configuration

Requirements:

1. Allow RELAYing for mydomain.com
2. Allow such relaying ONLY for hosts in the local network

Setup:

/etc/postfix/main.cf

inet_interfaces = all

 mynetworks = 10.0.0.0/16  (local network IP)

smtpd_recipient_restrictions =

        permit_sasl_authenticated

        check_recipient_access hash:/etc/postfix/recipient_access

        reject_unauth_destination

/etc/postfix/recipient_access:

mydomain.com   OK

sudo postmap /etc/postfix/recipient_access
sudo service postmap restart

kswapd0 high cpu consumption

OS: Ubuntu 15.10

Linux Kernel: 4.2.0-25-generic

Running MongoDB gets the CPU consumption high, with the culprit being kswapd0 process

Fix: 

add this to /etc/rc.local:

echo madvise > /sys/kernel/mm/transparent_hugepage/enabled

As per this URL: https://www.reddit.com/r/chrubuntu/comments/3ru04q/help_kernel_process_kswapd0_randomly_consuming/


The next attempt:

echo 3 > /proc/sys/vm/drop_caches

One more attempt:

worked around by commenting out line 2 in /lib/udev/rules.d/40-vm-hotadd.rules.

And the last attempt really worked:

Just updated to the last kernel update on ubuntu 15.10 :-)

Sending Google Docs Document as Email and Viewing in Gmail

Our system uses Google Docs to create and edit documents, which is used as email templates. When we need to send email, we fetch the Google Docs document as HTML (using gdata PHP/Python libraries). And this HTML contents is substituted with data from the database (aka merge fields -- I use smarty to accomplish) and this HTML contents is sent as html email.

All this is fine, but when the mail sent is viewed in Gmail (or probably Hotmail and other web-based email solutions), most HTML decorations are lost and the email is seen in Gmail mostly as text. However, when the same email is viewed using Thunderbird, the email retains all the HTML styles, decorations etc.

After some googling, the following was clear:

1. Gmail (and others) do not like CSS in the header, but they are somewhat okay with inline styles.
2. Some of the styles (like font-weight: bold, color within style tags ) etc are stripped out in Gmail loosing the HTML styles for their equivalents. There could be many such styles stripped out, I still don't know the logic of stripping styles like font-weight - but it happens.
3. Instead of such font-weight styles, they seem to like bold tags (<b>...</b>) etc.
4. Instead of color styles, they seem to like old font tags (<font color="#xxxxxx">....)
5. Google Docs sets up styles for a link (anchor) as color:inherit and text-decoration:inherit, which should rather not be there and the default behavior is preferred.

This is the minimum that I needed to get off, there could be various other styles and tags, that need to handled to show in Gmail properly.

The following script in PHP is helpful to move styles inline and then replace some styles with specific tags. 

function move_css_inline($text)
{
    $html = $text;
    if(!preg_match("/\

Add Javascript to Existing PDF files (Python)

There are several tools on the net, that helps you to add javascript to new PDF files - but I could not find any tool to add Javascript to existing PDF files.

Here is a python script that accomplishes that. For this you need to install the following python package (which is a modified pyPDF - found at http://pybrary.net/pyPdf/). The modified pyPDF is found at http://goo.gl/sJcMO. Or you could simply unzip the modified pyPDF in your current directory.


Help text and Usage for this script:

$ python addjs2pdf.py 
Usage: addjs2pdf.py [options] in-pdf-file out-pdf-file

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -j JAVASCRIPT, --javascript=JAVASCRIPT
                        javascript to embed (default embedded JavaScript is
                        app.alert messagebox)
  -f JAVASCRIPTFILE, --javascriptfile=JAVASCRIPTFILE
                        javascript file to embed

  add-js-to-pdf, use it to add embedded JavaScript to a PDF document that will execute automatically when the document is opened
  Based on modified pyPDF http://pybrary.net/pyPdf/ and inspiration from https://DidierStevens.com

$

Here is the addjstopdf.py:

from pyPdf import PdfFileWriter, PdfFileReader

import optparse

def Main():
    """add-js-to-pdf, use it to add embedded JavaScript to a PDF document that will execute automatically when the document is opened
    """

    parser = optparse.OptionParser(usage='usage: %prog [options] in-pdf-file out-pdf-file', version='%prog 0.1')
    parser.add_option('-j', '--javascript', help='javascript to embed (default embedded JavaScript is app.alert messagebox)')
    parser.add_option('-f', '--javascriptfile', help='javascript file to embed')
    (options, args) = parser.parse_args()

    if len(args) != 2:
        parser.print_help()
        print ''
        print '  add-js-to-pdf, use it to add embedded JavaScript to a PDF document that will execute automatically when the document is opened'
        print '  Based on modified pyPDF http://pybrary.net/pyPdf/ and inspiration from https://DidierStevens.com'
        print ''
        return

    input1 = PdfFileReader(file(args[0], "rb"))
    output = PdfFileWriter()
        
    pages = input1.getNumPages()
    for p in range(pages):
        output.addPage(input1.getPage(p))
    if options.javascript == None and options.javascriptfile == None:
            javascript = """app.alert({cMsg: 'Hello from PDF JavaScript', cTitle: 'Testing PDF JavaScript', nIcon: 3});"""
    elif options.javascript != None:
            javascript = options.javascript
    else:
        try:
            fileJavasScript = open(options.javascriptfile, 'rb')
        except:
            print "error opening file %s" % options.javascriptfile
            return

        try:
            javascript = fileJavasScript.read()
        except:
            print "error reading file %s" % options.javascriptfile
            return
        finally:
            fileJavasScript.close()

    output.addJS(javascript)
    outputStream = file(args[1], "wb")
    output.write(outputStream)
    outputStream.close()

if __name__ == '__main__':
    Main()

Setup Ubuntu server for PHP-MongoDB on Amazon AWS

This is a record of all the steps that I did to setup a Ubuntu server on Amazon AWS, primarily for PHP-Mongo setup with Apache2. I may consider nginx and fastcgi sometime later, but for now, will go ahead with Apache2.

AMI

AMI used is alestic-git-server-ubuntu-11.10-oneiric-amd64-20111017 (ami-dd73bfb4)

Packages Installed

Base Packages:

sudo apt-get -y install language-pack-en-base
sudo apt-get -y install unzip
sudo apt-get -y install python-setuptools
sudo apt-get install build-essential

PHP and Apache2:

sudo apt-get install apache2
sudo apt-get install php5
sudo apt-get install libapache2-mod-php5 php5-curl php5-gd php5-imap
sudo apt-get install php-pear php5-dev





MongoDB Install:

Steps as per http://www.mongodb.org/display/DOCS/Ubuntu+and+Debian+packages

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10

Append the following line to /etc/apt/sources.list:
deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen

sudo apt-get update
sudo apt-get install mongodb-10gen

Update (April 28th 2014): This has changed slightly. The updated steps below:

Steps as per: http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | sudo tee /etc/apt/sources.list.d/mongodb.list

sudo apt-get update
sudo apt-get install mongodb-org


PHP and Python Drivers for MongoDB:

sudo pecl install mongo
Add "extension=mongo.so" in /etc/php5/apache2/php.ini and /etc/php5/cli/php.ini

Update (April 28th 2014): The above adding to php.ini is changed for Ubuntu 14.04

echo 'extension=mongo.so' | sudo tee /etc/php5/mods-available/mongo.ini
sudo php5enmod mongo


sudo apt-get install python-dev

sudo easy_install pymongo

SunJava Install:

As per this link:

sudo apt-get install python-software-properties

sudo add-apt-repository ppa:ferramroberto/java

sudo apt-get update

sudo apt-get install sun-java6-jdk

Update (April 28th 2014): The above has changed for Ubuntu 14.04

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

SVN and other Tools:

sudo apt-get install subversion

sudo apt-get install libapache2-svn
sudo apt-get install git

sudo apt-get install mpack              # For MIME Pack/unpack

Specific Packages/Modules for my App:

sudo a2enmod rewrite
sudo a2enmod actions
sudo pear install --alldeps Mail
sudo pear channel-discover pear.pearplex.net
sudo pear install pearplex/PHPExcel

wkhtmltopdf:
Install http://code.google.com/p/wkhtmltopdf/  including wkhtmltopdf, wkhtmltoimage and libwkhtmltox  to /usr/local/bin, /usr/local/lib and /usr/local/include

Also install on Ubuntu libxrender, needed for wkhtmltopdf

sudo apt-get install libxrender-dev

PHP and Python bindings for libwkhtmltox:
git clone git://github.com/mreiferson/php-wkhtmltox.git   and follow README
git clone git://github.com/mreiferson/py-wkhtmltox.git    and sudo python setup.py install


Misc:

sudo apt-get -y install fontconfig xfs

sudo apt-get install libuuid1 uuid-dev uuid-runtime
sudo pecl install uuid

echo 'extension=uuid.so' | sudo tee /etc/php5/mods-available/uuid.ini
sudo php5enmod uuid


Others:

sudo apt-get install memcached php5-memcached
sudo apt-get install mcrypt php5-mcrypt
sudo php5enmod mcrypt


sudo apt-get install sqlite3 php5-sqlite
sudo apt-get install mysql-server php5-mysql

sudo apt-get install imagemagick

# Install Font?
sudo mkdir /usr/share/fonts/truetype/customttf
sudo cp /home/live/ereceipts/server/fonts/MTCORSVA.TTF /usr/share/fonts/truetype/customttf
sudo fc-cache -f -v


Other Edits:

1. php.ini in /etc/php5/apache2/php.ini (and the cli version)

error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT & ~E_NOTICE
date.timezone = Asia/Kolkata
session.gc_maxlifetime = 86400

Batch Convert Multipage CR2 to JPG

I landed up with a bunch of CR2 files (Canon Digital Raw Images from camera), which I needed to convert to JPG. On Ubuntu, I was able to view the CR2 format file using the installed Image Viewer and Gimp, after installing ufraw package.

However, the "convert" tool of ImageMagick was not able to convert to jpeg in a proper way - see the attached example converted images. This was due to the fact that the CR2 image has multiple pages and the "convert" tool is unable to understand this properly. (CR2 images with single page was successfully converted to jpg).

Gimp was able to open the CR2 file, but I had to specifically mention "Page1" in the "Import from Tiff"  dialog box. However, since I have to convert several CR2 files, manually opening them and converting was not an option. And I was also unable to figure out how to specify "Page1" in the gimp scripts (.scm files) or the Python-fu (pdb.file_load_tiff) scripts.

Finally, I was able to use the "convert" tool itself to convert to JPEG, but simply renaming the CR2 file as a TIFF file and using the same convert command like

convert renamed_cr2.tiff image.jpg

And that's it! It showed some errors about reading Tiff file (as it is not really a TIFF file) - but was able to successfully convert it.