Loading / reloading Pythonfilter

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Loading / reloading Pythonfilter

Alessandro Vesely
I changed the code of a filter which is already configured and running.  Now I
have to install it.  Last time I issued courierfilter stop, and then start.
However, that way I have to reload also clamav databases, which takes a long
time.  Is there any better way to upgrade the running Python process?

BTW, in order to tell distributed filter from home-brewed ones, I install sym
links in the dist-package/pythonfilter directory, which point to the source
file.  I note that Python does not produce any .pyc in that case.  However, the
following command compiles in the current directory irrespectively of whatever:

    python -m compileall -l attachments.py

Since the pyc is what should get loaded when loading a module, an idea could be
to reload modules if they have been recompiled since the last time they were
loaded.  Let me attach the idea.  Beware: I did'n run it, not even once, and
didn't fully understand the caveats about using reload(), see e.g.:

https://stackoverflow.com/questions/437589/how-do-i-unload-reload-a-python-module#answer-438845

One more question, about virtualenv.  Has anyone had experience loading
Pythonfilter that way under Courier?  Another possibility is to install
Pythonfilter globally, but keep additional packages in a virtualenv, just to
ease tracking requirements.

TIA
Ale


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
courier-users mailing list
[hidden email]
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-users

pythonfilter-idea-for-reloading.diff.txt (2K) Download Attachment
signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Loading / reloading Pythonfilter

Gordon Messmer-2
On 06/07/2017 04:32 AM, Alessandro Vesely wrote:
> Last time I issued courierfilter stop, and then start.
> However, that way I have to reload also clamav databases, which takes a long
> time.  Is there any better way to upgrade the running Python process?

That should only be the case if you were running pyclamav, which is no
longer available or supported.  The supported means of using clamav is
to run an instance of clamd as the courier user, and use pyclamd to scan
messages.  In that configuration, you should be able to "filterctl stop
pythonfilter/start pythonfilter" without significant delays.

> BTW, in order to tell distributed filter from home-brewed ones, I install sym
> links in the dist-package/pythonfilter directory, which point to the source
> file.  I note that Python does not produce any .pyc in that case.

That's true, but that only means that python will take slightly longer
to parse those files when they're imported.  Since that only happens on
startup, it's hard to make the case that producing pyc files will
significantly impact performance at all.

> Since the pyc is what should get loaded when loading a module, an idea could be
> to reload modules if they have been recompiled since the last time they were
> loaded.  Let me attach the idea.  Beware: I did'n run it, not even once, and
> didn't fully understand the caveats about using reload(), see e.g.:
>
> https://stackoverflow.com/questions/437589/how-do-i-unload-reload-a-python-module#answer-438845

pythonfilter uses threads in order to allow modules to keep statistics
in memory.  As far as I can tell, reloading isn't going to be safe.

> One more question, about virtualenv.  Has anyone had experience loading
> Pythonfilter that way under Courier?  Another possibility is to install
> Pythonfilter globally, but keep additional packages in a virtualenv, just to
> ease tracking requirements.

I haven't, personally.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
courier-users mailing list
[hidden email]
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Loading / reloading Pythonfilter

Alessandro Vesely
On Wed 07/Jun/2017 21:39:34 +0200 Gordon Messmer wrote:

> On 06/07/2017 04:32 AM, Alessandro Vesely wrote:
>> Last time I issued courierfilter stop, and then start.
>> However, that way I have to reload also clamav databases, which takes a long
>> time.  Is there any better way to upgrade the running Python process?
>
> That should only be the case if you were running pyclamav, which is no longer
> available or supported.  The supported means of using clamav is to run an
> instance of clamd as the courier user, and use pyclamd to scan messages.  In
> that configuration, you should be able to "filterctl stop pythonfilter/start
> pythonfilter" without significant delays.
Hm... I wouldn't recommend that.  During the 17 seconds it took to reload "C"
filter databases, I only had one "432 Mail filters temporarily unavailable".
Of course, stopping and restarting pythonfilter alone would have taken much
less.  Short as that time might have been, messages with bad attachments would
have slipped through unfiltered in the meanwhile.

Ale
--



















------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
courier-users mailing list
[hidden email]
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-users

signature.asc (484 bytes) Download Attachment
Loading...