Mar 24
    Change language to :

So, Saturday and today I spend a bit of time updating my honeynet check mirror application, which retrieves information from various mirrors to do an educated guess on when the mirrors did a last sync (all in perspective ofcourse because it doesn’t tell the actual status, but at least it’s a guide, and if properly done, we can see what happends).

That ofcourse doesn’t mean the tool is honeynet only, you can easily change it for your own needs to enlist several own mirrors etc. It’s just that I named it for the honeynet project: check_honeynet, one could argue to name it “evil-mirror-validator” or something (evil- are prefixes of most of my projects).

Anyway: back to the discussion: v1.7 has a new way of parsing configuration records which is rather straight forward. I will be implementing the same for 2.0 (which is actually just a continued development on top of 1.7). This makes use of the Config::Simple module, which enables me to write it all ini-style based (it was already like that, but then resolved more ugly then it is now).

If you browse to the downloads section, you will find it there, with a published web-frontend and so on. If you have any comments please let me know. From what I saw from my local tests this just works fine.

For 2.0 I will be issueing a seperated statistics file, which I will write ini based (from the application) to gather statistics about [ok] [bad] [banned] hosts. This way we can more selectively send messages to the bogus mirror, and even ban them (which could result in an automated removal from the main site for example) if they trigger the (adjustable) thresholds.

written by Remko \\ tags: , , , , ,

Mar 20
    Change language to :

Hello,

It was a time ago already that I posted some updates to the site, I had been very busy with work and things around that, so E_NOTIME to bring in updates. Because I dont see that change anytime soon, I’ll stick with some Technical Contributions to the internet. I am playing a lot with Checkpoint at the moment, and I would like to document some trivial things, as well as some other things that are not as trivial as they seem. I know that others can use the information (hence even Elsevier contacted me recently to help write a book about Checkpoint NGX) so lets try to share and document it properly.

In the menubar you can see that I moved a lot of technical items to the “Technical” menu, I’ll place nice information there. I will probably bring in a seperated category for Checkpoint under the FAQ as well so that it’s even easier to spot.

I am also writing up more for FreeBSD in the near future (if time permits) and I am pondering in documenting a few PoC’s that I did recently.

So expect information back on the board soon, but not regarding my private life anymore, I just dont have enough time to do so.

written by Remko

Mar 20
    Change language to :

Allthough I am a die-hard FreeBSD enthusiast, lets congratulate the NetBSD group with it’s 15th anniversary!

FreeBSD and NetBSD always have had a friendly competition with regards to implementing new technology, I hope we can both learn and adopt from that for at least the next 15 years :-)

For more information about the NetBSD Operating System: www.netbsd.org
For more information about the FreeBSD Operating System: www.freebsd.org

written by Remko

Dec 22
    Change language to :

There could be times that BGP is not being synchronised properly or something. First of all, check up the localnetmask on both routers. I have -seen- issues that the netmasks weren’t the same in the same network which prevented BGP from properly being exchanged. So, hint: Check netmasks / network information and make sure they are correct!

written by Remko

Nov 28
    Change language to :

So, today I "updated" wordpress to match my digital camera. Why did I do this? Well easy! :-) Currently wordpress supports thumbnails for uploaded pictures if the file is <= 3 megapixel in size. Not megabytes or something, megapixel.

I updated the wp-admin/includes/image.php file to be able to upload files (and thumbnail them) if they are bigger then that size. This way I can normally add my digital photos that were created by my Sony Alpha 100 camera.

What did I do?

Edit wp-admin/includes/image.php and search/replace the following line (at the moment of writing this is line 150 with wordpress 2.3.1)

CODE:
  1. $max = apply_filters( 'wp_thumbnail_creation_size_limit', 3 * 1024 * 1024, $attachment_id, $file );

with the following line:

CODE:
  1. $max = apply_filters( 'wp_thumbnail_creation_size_limit', <amount of megapixels you want to support here> * 1024 * 1024, $attachment_id, $file );

EDIT: Please take notice that you might run into server errors, because of PHP running out of it's allowed memory size. You can set these values in ``php.ini``.

written by Remko

Oct 05
    Change language to :

Today I was toying around with Eventia Reporter in order to generate some reports about the things I did with checkpoint etc. Since I build up a new environment from scratch I needed to import various logfiles. When you enter the GUI and start a recording session there ; you will find that you can only do this one log at a time. Which is time consuming and frustrating because the machines have time available to process logfiles instead of having free-cpu cylcles left. I couldnt find any usefull option last night when I tried to do this scriptified; but I found some information on the CPUG.org forum that could help you (original post here ).If you want to do this automatically you should do something like:
log_consolidator -R -e Yes -s ip-addr-of-logserver -x Yes -o No -t Specified_Log -l <logfile>.log -a Begin_of_Log -b <your connections table>

If you script that it would become something like (easy example on splat, use nohup to start the process or you will loose the script eventually due to automatic logouts):


#!/bin/sh

APP=/opt/CPrt-R65/log_consolidator_engine/bin/log_consolidator
LOGDIR=/opt/CPsuite-R65/fw1/log # use the path where your backup logs are stored here

for LOG in `ls $LOGDIR/*<pattern if needed>*.log`
do
$APP -R -e Yes -s <loghost> -x Yes -o No -t Specified_Log -l $LOG -a Begin_of_Log -b <table>
done

written by Remko

Oct 02
    Change language to :

Lately I have been attempting to upgrade several Checkpoint Smart Center stations from R60 to R65. Not without a glitch though. So I thought lets write a little bit of documentation about this.

Initially I needed to upgrade the old management station twice to get to R65 which already was a pitfall. But after that it started working properly.Next I downloaded the latest NGX utilties on the internet and performed a backup to prepare for an OS upgrade (changing OS from the company in Redmond to SPLAT).

So far everything went smooth and I felt really relaxed doing it.I rebuild one of the management stations with SPLAT R65 and copied over the huge file (2.5GB in size) which took a little and yeah it was finally there. I wanted to import the file, but sadly I got several errors stating that I could not copy the file etc. I ofcourse am ignorant so I tried to do this manually; and darn that works! Now extracting the file... also works. So why does "upgrade_import" cause problems? After a long struggle, even noticing (I posted forum posts) this on the internet I found the cause of the problem at least the problem for me.

My backups were huge because of database revisions being saved in it as well (as it should) I removed them after a tip from my support company (the tip was for something else but it lead to the solution) and remained with a 53MB file (yeah a bit smaller then before ;-)).

The moment that my support party told me that I could lower the size of the file so that we could analyze it together I got struck by lightning (well an idea struck me) What if the system does not allow files bigger then 2GB (Which is probably hardcoded because the manual copy went OK!) ? So now having the 53MB big backup file, I copied that over much quicker and imported that... it worked..

Symptoms:

DecompressImportedFile: Error >> Failed to copy /var/backups/xxxxx.tgz to /opt/CPsuiteR65/fw1/tmp/upgrade_temp_dirSolution:

Get rid of the database revision files (You can safely back them up and remove the contents of the repository directory and the fwdatabaserevision.dwb file (or something like that)) rerun the backup import on that specific backup and be done with it.

written by Remko

Aug 01
    Change language to :

For long I used a little script called logtransfer, which is just a stupid script that scp's files over *2007-07-01* for example, but that wasn't very flexible anymore. I setup rsync with some modules that enable me to transfer the logfiles in an almost similiar setup as before, but now with partial filetransfer support (continue where the beast stopped, to prevent unneeded transfers), deletion support (that if files are gone, they are removed locally as well, I wont do that before I made backups on DVD ofcourse).

This generates much more flexibility for me, because filenames can change, things can get deleted by hand, etc and that is taken into account now (I realise that this is not something everyone want to persue, but I do :)).

Backups are now in quicker; more complete and more well, my stylish. No more need to maintain scripts myself; but just use standard tools already available that do the job very well.

(Oh, ofcourse I use rsync for much more then just my backups and logfiletransfers, I use it for the Honeynet project, mirroring my webtrees to an offsite location, making sure that my patches on the FreeBSD cluster are synchronised with my local tree etc).

Example rsync usage that I do  now:

/usr/local/bin/rsync --ignore-errors -av --partial --delete remotehost::modulename targetdirectory

Continue reading »

written by Remko

Jul 30
    Change language to :

And another version had been released; 1.6.2. This contains updates to the mailer code, from Net::SMTP to Mail::Sendmail (as suggested by Richard Arends <richard at unixguru dot nl> (one of my collegues)). This made the script capable of sending out emails in case there were problems, which didn't work very well in the previous release. Well it did at my host, but apparently Lance' host got into troubles with this. Please test and report to me when things are not working. You can find the latest script here. Continue reading »

written by Remko

Jul 30
    Change language to :

Well well, another quick update following shortly after the previous one. It appears that the previous setup was not good enough to warrant the release actually. There were some flaws in 1.6.1 that prevented the mail option from being used. Lance reported these errors to me and I could not easily fix them. (I thought I did well but apparently not). I now decided to take the alternative route, already suggested by Mart vd Wege in 2006;
use a different Perl module to sendout the emails; and wunderbarr that appears to be working out of the box. At least on my machine.

So I did something evil; normally this should have been delivered to 1.7.0 instead of 1.6.2 but, since it is a bug and I wish to do other things for 1.7.0 or perhaps even 2.0.0 I implemented this early on in the current game (it still follows the new tradition for version numbering etc). For this to work you actually need the following script and configuration file which can be used to well checkout mirrors :-).

If you encounter any issue please let me know!

PERL:
  1. #!/usr/bin/perl
  2. ###########################################################################
  3. # $Id: check_honeynet.pl,v 1.45 2007/07/30 16:36:00 remko Exp $
  4. ###########################################################################
  5.  
  6. ###########################################################################
  7. # Copyright (C) 2005-2007, Remko Lodder <remko at FreeBSD dot org>. All rights reserved.
  8. #
  9. # Redistribution and use in source and binary forms, with or without
  10. # modification, are permitted provided that the following conditions
  11. # are met:
  12. # 1. Redistributions of source code must retain the above copyright
  13. #    notice, this list of conditions and the following disclaimer.
  14. # 2. Redistributions in binary form must reproduce the above copyright
  15. #    notice, this list of conditions and the following disclaimer in the
  16. #    documentation and/or other materials provided with the distribution.
  17. #
  18. # THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS ``AS IS'' AND
  19. # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
  20. # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
  21. # ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
  22. # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
  23. # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
  24. # OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
  25. # HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
  26. # LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
  27. # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
  28. # SUCH DAMAGE.
  29. ###########################################################################
  30.  
  31. ###########################################################################
  32. # Contributors:
  33. # Ivo Naninck, (Language)
  34. # Marc Plaisier, (Language)
  35. # Mart van de Wege, (Perl mailer suggestion)
  36. # Lance Spitzner (the most valueable feedback regarding the script).
  37. #
  38. # Script:
  39. # This is a stand alone script written for honeynet.org. It's purpose is
  40. # to check the listed mirrors to see which one is outdated and notify
  41. # the administrator of the outdated mirror if needed. This way the
  42. # mirrors will always be within a certain timeframe and can be removed
  43. # if they are too outdated.
  44. #
  45. # A while ago Mart vd Wege told me that a higher level mail module would be
  46. # much easier to cope with. This had been implemented as of version 1.6.2
  47. #
  48. # Written in colaboration with Lance Spitzner <lance at honeynet dot org>
  49. ###########################################################################
  50.  
  51. ###########################################################################
  52. # Differences between releases, started this between 1.5 and 1.6 sorry
  53. # for the loss of potential usefull information (although I might be able
  54. # to retrieve the missing data from my CVS branches).
  55. # Note that all the three version based versions are just bugfixes to some
  56. # degree.
  57. # Version       Who             What
  58. # 1.1           Remko           Initial script to check the mirrors.
  59. # 1.1->1.2      Remko
  60. # 1.2->1.3      Remko
  61. # 1.3->1.4      Remko
  62. # 1.4->1.5      Remko           Code cleanups, restructure of code; corrected
  63. #                               some bugs between various releases spotted
  64. #                               by Lance.
  65. # 1.5->1.6      Remko           Cleanups, changed structure for timestamp
  66. #                               retrieval; make it human readable and match
  67. #                               on a specific pattern. Made the version dependend
  68. #                               on the configuration version and visa-versa
  69. #                               to be able to make big config changes.
  70. #                               BF-1: Fix the email send options by using the
  71. #                                     correct Net::SMTP commands.
  72. #                               BF-1: Fix the parsing of the new mirrorprobe
  73. #                                     layout.
  74. #                               BF-2: Change the Mailer used to send out the
  75. #                               report.
  76. ###########################################################################
  77.  
  78. use strict;
  79. use warnings;
  80. use LWP::Simple;
  81. use Getopt::Std;
  82. use Mail::Sendmail;
  83.  
  84. ###########################################################################
  85. # variables. All configurable options are defined below. Please adjust them
  86. # to your need.
  87. ###########################################################################
  88.  
  89. # create three hashes which can be used to read our config variables from the
  90. # configuration file. Devided between version info, configuration items and mirror     
  91. # information.
  92.  
  93. my %VERSION             = ();
  94. my %CONFIG              = ();
  95. my %MIRRORS             = ();
  96.  
  97. # create the option hash, we will use that later to add our option flags.
  98. my %option              = ();
  99.  
  100. # Template some standard variables
  101. my ($CONFIGFILE, $status, $reporthost, $timestamp,$target);
  102.  
  103. # Prototype functions
  104. sub read_conf ($);
  105. sub fetch_data ($$);
  106. sub email_report ($$);
  107. sub convert_input ($);
  108. sub process_mirrors (%);
  109. sub create_timestamp ($);
  110.  
  111. # Version, author and script specific behaviour
  112. my $author              = 'Remko Lodder <remko at FreeBSD dot org>';    # Name of the author, respect the license.
  113. my $name                = __FILE__;                             # Our scriptname.
  114. my $MAJOR               = '1';                                  # Our major version number (ma.mi.pa)
  115. my $MINOR               = '6';                                  # Our minor version number
  116. my $PATCHLEVEL          = '2';                                  # Our patchlevel
  117. my $version             = "$MAJOR.$MINOR.$PATCHLEVEL";          # Our version.
  118.  
  119. ###########################################################################
  120. # Do not edit anything below this line unless you know what you are doing.
  121. ###########################################################################
  122.  
  123. getopts("cC:f:F:hpR:t", \%option);
  124.  
  125. my $config_flag   = 1 if $option{f};
  126. my $config_option = $option{f};
  127.  
  128. my $create_flag   = 1 if $option{c};
  129. my $help_flag     = 1 if $option{h};
  130. my $process_flag  = 1 if $option{p};
  131. my $test_flag     = 1 if $option{t};
  132. my $convert_flag  = 1 if $option{C};
  133. my $fetch_flag    = 1 if $option{F};
  134. my $retrieve_flag = 1 if $option{R};
  135.  
  136. # Simply assign the contents of the configuration parameter to the
  137. # configuration file variable, otherwise overrule it with the
  138. # default value
  139.  
  140. $CONFIGFILE = $config_option || "./honeynet.cf";
  141. &read_conf($CONFIGFILE);
  142.  
  143. # Make sure that there us a version statement in the configuration file, so that we can see whether we are
  144. # compabible or not.
  145.  
  146. if (!$VERSION{'version'})
  147. {
  148.         print "It appears that you do not have a version statement in your configuration file. This means that the version you are using now is
  149. too old, make sure that you obtain the latest one and update that to your needs.";
  150.         exit(1);
  151. }
  152.  
  153. if ($VERSION{'version'})
  154. {
  155.         my($version_def_major,$version_def_minor) = $VERSION{'version'} =~ /(\d+)\.(\d+)/;
  156.  
  157.         # the script can defer between MAJOR and MINOR releases; bugfixes aka patchlevels are
  158.         # not affected by this and can thus be ignored.
  159.         # TODO: This check should be made more flexible in the future, version 1.4 and 1.5 share
  160.         # the same configuration file and should both pass.
  161.  
  162.         if(($version_def_major lt $MAJOR) or ($version_def_minor lt $MINOR))
  163.         {
  164.                 print "You appear to be using an older configuration file that might not be compatible with the current version of the
  165. script, please validate that you have the latest options included and copy over the version statement from the latest available configuration
  166. file. Make sure that the old version statement is overwritten!
  167. The current script runs on version: $version, while the configuration is for version $VERSION{'version'}\n";
  168.                 exit(1);
  169.         }
  170. }
  171.  
  172. # Create a new timestamp that will be fed into the mirrors, which we can use to test the
  173. # age of the mirror.
  174.  
  175. if ($create_flag) {
  176.         create_timestamp($CONFIG{probefile});
  177. }
  178.  
  179. elsif ($process_flag) {
  180.         # Process the mirrors using the hash we have for them.
  181.         process_mirrors(%MIRRORS);
  182. }
  183.  
  184. # test mode, printout information on screen.
  185. elsif($test_flag)
  186. {
  187.         # In test mode we dont send out emails.
  188.         $CONFIG{'enable_mail'} = 0;
  189.  
  190.         print("$name: Starting\n");
  191.         print("$name: Processing mirrors\n");
  192.  
  193.         # Process the mirrors using the hash we have for them.
  194.         process_mirrors(%MIRRORS);
  195.  
  196.         print("$name: Finishing\n");
  197. }
  198.  
  199. # convert input from unixtime to human readable time.
  200. elsif($convert_flag)
  201. {
  202.         print "$option{C} resolves to " . convert_input($option{C}) . "\n";
  203. }
  204.  
  205. # fetch the mirrorprobe file from the given host
  206. elsif($fetch_flag)
  207. {
  208.         my $result = fetch_data($option{F}, $CONFIG{'sourcefile'});
  209.         open(OUT, "> $CONFIG{'outdir'}/$option{F}.timestamp");
  210.                 print OUT $result;
  211.         close(OUT);
  212. }
  213.  
  214. # Fetch the mirror timestamp and parse it. Print the output back on the screen.
  215. elsif($retrieve_flag)
  216. {
  217.         my $result = fetch_data($option{R}, $CONFIG{'sourcefile'});
  218.         print("$option{R} was last modified " . convert_input($result) . "\n");
  219. }
  220.  
  221. # People expect a help option, provide it for them.
  222. elsif ($help_flag)
  223. {
  224.         print_help();
  225. }
  226.  
  227. # No valid options had been given, fallback to the help information.
  228. else
  229. {
  230.         print_help();
  231. }
  232.  
  233. sub print_help
  234. {
  235.         print("Usage:\t$name [-c] [-C <value>] [-f <configurationfile>] [-F <host>] [-h] [-p] [-R <host>] [-t]
  236. \t-c\tCreate the timestamp for the localmachine. This timestamp can be used to determine when the mirror was last updated.
  237. \t-C\t<value> converts the unix timestamp to human readable format
  238. \t-f\t<filename> Use the specified configuration file
  239. \t-F\t<host> fetch the timestamp for an external host, for example: www.honeynet.nl
  240. \t-h\tprint this help.
  241. \t-p\tCheck the status of the mirrors, and report the output to us
  242. \t-t\tTest mode, does not send out emails, but prints the information on the screen.
  243. Version: $version
  244. Originally written by Remko Lodder <remko\@FreeBSD.org, for the honeynet project.\n");
  245. }
  246.  
  247. sub create_timestamp ($)
  248. {
  249.         my $probefile = shift;
  250.         open(F_OUT, "> $probefile");
  251.                 print F_OUT "Mirrorprobe time: " . time() . "
  252. Local time: " . convert_input(time());
  253.         close F_OUT;
  254. }
  255.  
  256. sub convert_input ($)
  257. {
  258.         my $output      = scalar localtime(shift);
  259.         return $output;
  260. }
  261.  
  262. sub fetch_data ($$)
  263. {
  264.         my $source      = shift;
  265.         my $sourcefile  = shift;
  266.  
  267.         my $return_data = ();
  268.         my $data        = get("http://$source$sourcefile");
  269.  
  270.         # IF the remote data is present, take out the numberic time value and return that
  271.         # ELSE obscure the data, which will revert to 1969/1970 (depending on machinetime).
  272.         if ($data)
  273.         {
  274.                 chomp $data;
  275.  
  276.                 # TODO: Compatibility requirement till mirrors are on 1.6
  277.                 if ($data =~ /^(\d+)/)
  278.                 {
  279.                         return $data;
  280.                 }
  281.                 else
  282.                 {
  283.                         $return_data = $data;
  284.                         $return_data =~ s/\n/\ /;
  285.                         if ($return_data =~ /\S+ \S+ (\d+) \S+/)
  286.                         {
  287.                                 $return_data = $1;
  288.                         }
  289.                         return $return_data;
  290.                 }
  291.         }
  292.