Tag Archives: python

[python] Improve performance of application-logging ( Part 1 – server side)

Everybody knows that logging something on screen or to a file is inevitable when you create program. It makes thing visible to you , easy control and for debugging also.

But recently what I have faced is that the performance of my application ( a bunch of services written in python) has decreased so much when I try to print out so many thing to the console ( the primary I/O channel I have ever used).

So I’m finding some ways to make this process become better.

About the context, these log is actually not realtime-critical. I want want to know which step my program is at and what is the result of the executing. That’s all.

The first testing method I try to implement is using UDP broadcasting . Sound so much insanity , right ? I just want to separate the logging process to other task. Not on my main task. And also, that logging should be flexible enough that I can choose to write on the console, write events to database or to the file without any performance-losing from my main jobs.

I have already installed twisted on my machine. So I will take advantage of it.

Let’s build a UDP Event logger server, just modify the class name of UDPEchoServer example.

We have:
from twisted.internet.protocol import DatagramProtocol
from twisted.internet import reactor
from pprint import pprint
class UDPLogger(DatagramProtocol):
 def datagramReceived(self, datagram, address):
 print 'signal from '
 pprint(address)
 print 'content of msg = ' + repr(datagram)
 self.transport.write(datagram, address)
def main():
 reactor.listenUDP(8000, UDPLogger())
 reactor.run()
if __name__ == '__main__':
 main()

[Ubuntu] Install Scrapy in Ubuntu 12.04/12.10 with easy_install

– I have tried to install Scrapy with easy_install ( install easy_install_tools)
– then when running easy_install Scrapy .
it got stuck at step installing twisted with some fucking error like

twisted/runner/portmap.c:10:20: fatal error: Python.h: No such file or directory
compilation terminated.

it turned out I had to install python-dev packge first
( ref: http://nickhumphreyit.blogspot.com/2011/10/buildbot-install-error.html )


apt-get install python-dev


[Bug-Database]Problems when installing Twisted Framework.

Platform: Ubuntu 11.04 as Based-Version ( LinuxMint 12 – Lisa Distro)

Native Compiler : gcc

Python 2.7.2 +  ( python –version)

Twisted Framework here:  http://twistedmatrix.com/trac/wiki/Downloads

Download, Extract, and run :

python setup.py install

Error received :  

twisted/runner/portmap.c:10:20: fatal error: Python.h: No such file or directorycompilation terminated.

Fix : install package python-dev  

sudo apt-get install python-dev

Problem fixed !