Recent Updates Toggle Comment Threads | Keyboard Shortcuts

  • danw 2:53 pm on 2015-09-17 Permalink | Reply
    Tags: cloud, on-prem, stupid   

    Cloud vs on-prem 

    Today I read this stupid comment on Slashdot equating the “cloud* vs on-prem” debate to the historical “horse and buggy vs. auto-mobile” debate and had to object to stupidity. (* note: for the purposes of this post “Cloud services” assumes IAAS. You get network and VMs, you may or may not install your own OS, but the service provider does nothing beyond patch a vanilla OS install)

    >In reality a motorised carriage is more expensive and less safe in almost all cases….

    This is a stupid analogy. A much better one is purchasing a car vs. taxi cab. Short-term a taxi is much cheaper, and the primary reason I don’t purchase a car to use for during my vacation. Depending on your usage levels it can be cheaper even long term. Many people who live in large metropolitan cities are quite happy without cars and rely on taxis for their short-term car needs. It is a costs-benefits analysis.

    Cloud services vs. on-premises is the same. If you stop to think it is very obvious. The ONLY reason cloud can be cheaper is because the cloud service can have:

    1. Higher density in the data center which results in cheaper per server cost of space (fewer half full racks)
    2. Better hardware discounts due to greater purchasing power (they have your purchasing power as well as all their other customers purchasing power)
    3. Better labour utilization due to larger and more diverse workload (they have enough work to justify hiring a specialist and optimize their work assignments better)
    4. Better economies of scale in terms of quality of hardware purchased (eg. 36x HP DL360 vs 5x HP DL580)

    Those savings, plus a healthy margin, have to save you enough money to off-set:

    1. The cost of an Improved network link to the cloud provider (You DO want to communicate with those servers right?)
    2. Increased uncertainty of heavily relying on the cloud provider (Are they paying their bills? Are they going to warn you if they lost backups of your data?)
    3. The cost of migrating to cloud service (This will probably cost as much as your first year of cloud service)
    4. Security (Talk to a security consultant or three about cloud)
    5. Loss of flexibility to cloud service “standards and procedures” (Don’t want any change while you diagnose an issue? You have an issue with the latest version of ESXi? Solaris VMs don’t work well with KVM? Tough!)

    So it basically comes down to a cost-benefits analysis. And it turns out that many of the early adopters had unrealistic expectations and are now migrating back to on-premises. Economies of scale turn in to diminishing returns very quickly in the data centre world. If you have a very generic service need and your DC+hardware+OS licensing budget is over $300,000/year you are not going to save significant amounts of money going to the cloud and will be adding significant risk.

    Many managers want to use “cloud” simply to remove it from their area of responsibility but most of the time that is not a good enough reason when weighed against the inflexibility cloud introduces. And with all things, the better and more flexible a cloud service you get, the greater the cost of going to the cloud.

    Some organizations can save drastic amounts of money going to “the cloud”. Others would waste their money to an equal degree. As many are prone to point out when you replace “the cloud” with “someone else’s servers” the bloom comes off the rose pretty quick.

  • danw 3:59 pm on 2015-06-14 Permalink | Reply
    Tags: passwd useradd script   

    Script to convert /etc/passwd to useradd command line 

    This is a very simple script to convert a /etc/passwd file to the associated useradd command line that creates the account. Useful for when your brain has turned off and you’re mid migration to a new system. Works with Linux and Solaris.

    #!/usr/bin/perl -w
    use strict;
    while(<>) {
      my @a = split /:/;
      print 'useradd ';
      print '-d ' . $a[5] . ' ' if(defined $a[5]);
      print '-m -u ' . $a[2] . ' -g ' . $a[3] . ' ' ;
      print '-c "' . $a[4] . '"' if(length $a[4] > 0);
      if (!defined $a[6]) {
        $a[6] = "/usr/bin/false";
      print ' -s ' . $a[6] . ' ' . $a[0] . "\n";
  • danw 5:06 pm on 2014-05-30 Permalink | Reply  

    ZFS issues! 

    I’ve been having troubles with writes to my home ZFS storage. Writes seem to block, sometimes for hours, but never generate an error. I don’t get messages in dmesg and the per disk error counters dont increment.

    I went on the zfs mailling list, and first recommendation was to check if the zpool was suspended:

    root@Gangsta:~# echo "::walk spa | ::print spa_t spa_name spa_suspended" | mdb -k 
    spa_name = [ "rpool" ]
    spa_suspended = 0
    spa_name = [ "tank" ]
    spa_suspended = 0

    Second step was to see where the threads were stopping

    root@Gangsta:~# echo "::stacks -m zfs" | mdb -kw
    THREAD           STATE    SOBJ                COUNT
    ffffff00171eac40 SLEEP    CV                      2
    ffffff0017796c40 SLEEP    CV                      2
    ffffff001713dc40 SLEEP    CV                      1
    ffffff0017143c40 SLEEP    CV                      1
    ffffff001779fc40 SLEEP    CV                      1
    ffffff0017424c40 SLEEP    CV                      1
    ffffff0566af1820 ONPROC                     1

    Most recently I’ve been asked to:

    echo "::stacks -c spa_sync" | mdb -k
    Also can you please include 'zpool iostat tank 1' collected over a this period?

    So here is the layout of the disks

    root@Gangsta:~# zpool status tank
      pool: tank
     state: ONLINE
      scan: scrub repaired 103K in 106h36m with 0 errors on Fri May 23 22:36:19 2014
            NAME         STATE     READ WRITE CKSUM
            tank         ONLINE       0     0     0
              raidz2-0   ONLINE       0     0     0
                c6t16d0  ONLINE       0     0     0
                c6t20d0  ONLINE       0     0     0
                c6t24d0  ONLINE       0     0     0
                c6t28d0  ONLINE       0     0     0
                c6t55d0  ONLINE       0     0     0
                c6t53d0  ONLINE       0     0     0
                c6t49d0  ONLINE       0     0     0
                c6t51d0  ONLINE       0     0     0
                c6t54d0  ONLINE       0     0     0
                c6t56d0  ONLINE       0     0     0
                c6t52d0  ONLINE       0     0     0
                c6t50d0  ONLINE       0     0     0
              c6t26d0s0  ONLINE       0     0     0
              c6t30d0s0  ONLINE       0     0     0
              c6t26d0s1  ONLINE       0     0     0
              c6t30d0s1  ONLINE       0     0     0
              c6t18d0    AVAIL   
              c6t22d0    AVAIL   
    errors: No known data errors

    Output from iostat

    Output from mdb

    Output from zpool iostat

  • danw 6:22 pm on 2014-02-03 Permalink | Reply
    Tags: ,   

    3D printer arrived 

    The nifty 3d printer from DIY Tech Shop finally arrived! It’s been a long row to hoe.

    I originally ordered from MakiBox last September (2013), but they are/were having serious production issues; if you want to talk to some seriously annoyed people check out the forum. After the Nth delay (N is >= 4) in mid December 2013 I gave up on MakiBox and cancelled my order, even after they contacted me an assured me that my order was going to be “coming real soon now.” I ordered from DIY Tech Shop, although their website stated that they had run out of parts, and were planning on a large buy in early/mid February 2014. It arrived on Tuesday January 28th, 2014. As of this writing, Monday February 3rd, 2014, MakiBox is still only showing 504 shipped units. I am sure the forum has even more angry people asking the operator of MakiBox to remove his wildly optimistic “shipping estimates”, where by “shipping” they mean “lie” and by “estimates” they mean “LIE!”.

    So it arrived and I’ve gotten the hardware assembly done (see below).

    My oberservation:

    • The brackets for the Z-axis motors were not designed for the shape of these motors, lacking clearance for a ring near the output shaft. The first bracket cracked before I noticed the issue, the second one I shaved by hand to fit the motor
    • A few of the brackets pay more attention to aesthetics than function, lacking some more useful functionality. E.g. left and right sided parts that are functionally interchangeable, but would benefit from having nuts on either side to better fine tune connections.
    • It looks like the threaded rod was not cut with a chop saw, and one end of one of the rods wouldn’t let a nut on. I cleaned it up with a die, but I probably could have made do by spinning a nut down the rod from the good end and using a jam-nut to force it through to clean up the threads.
    • The website advertises a “RAMPS 1.4″ Arduino shield to manage the printer, but the manual specifies annother name. Although aparently they are the same board it confused me, in addition the RAMPS 1.4 board seems to be a dual hot-end controller and the manual doesn’t specify which parts are for the first hot-end and which are for the second hot-end. And since I only have ONE hot-end…..

    Now I need to start on the electronics. It’s just a bunch of crimp-on butt-joints but I’m not super confident of the vary non-specific nature of the instructions.



    • Finish wiring
    • Get filiment
    • Print enough replacement brackets for a whole printer
    • Get components for a second hot-end
    • Redesign to a Bowden type extruder
    • Start work on a 3d printer that can print parts at least as large as the width of a rack server
    • Design brackets for the backplanes
    • Kevtronic 1:23 pm on 2014-02-05 Permalink | Reply

      Uhhh.. huh huh… you said crimp-on butt-joints… Uhhhh…. huh huh….

    • WiggleChampion 1:46 pm on 2014-02-05 Permalink | Reply

      cool. very very cool.

  • danw 8:00 pm on 2014-01-09 Permalink | Reply
    Tags: ,   

    Storage pod backplanes arrived 

    So the backplanes arrived. They look good. Upon investigation, it turns out these have SAS1 (3gbps) SAS expanders, and can only recognize up to 2048GB of disk, so when 3TB SFF drive appear on the market they will not be able to accommodate them.

    Now I need to make a CAD representation of the PCB so I can design the bracket I want to 3D print to hold the backplane, drives and fans. Hopefully I can do a reasonable unit with easy removal for on site servicing.

    front of backplane

    Front of backplane

    back of backplane

    Back of backplane

    Apparently my work BB takes poor pictures.

  • danw 4:16 pm on 2014-01-06 Permalink | Reply
    Tags: , ,   

    Current storage setup 

    I’ve sunk a lot of $$$ into storage over the years and here is my current setup.

    • IBM x3650
    • 12GB ram
    • 2x dual core 3.0Ghz Xeon
    • 2x Fibre Channel PCIe cards
    • 5x SGI SE3016 disk trays (16 disks, 3u rack mount) (here)
    • 64x 40GB SATA disks for high performance I/O
    • 14x 2TB WD RED disks for bulk storage
    • 2x 120GB SSD disks for ZFS cache

    This allows me to host 20+TB of long term storage on the bulk storage, and 2TB for performance I/O, and export ZFS duplicated LUNs via redundant fibre channel switches.

    Future major additions are:

    1. Modify SE3016 disk trays to use SAS2 expanders that support 3TB and larger disks
    2. Attach bulk 2.5″ SFF SSD and SAS disks to storage server via new custom made disk tray (here)
  • danw 3:25 pm on 2014-01-06 Permalink | Reply
    Tags: ,   

    starting a delta type 3d printer 

    I’ve ordered a 3d printer (here) and I’ve decided I’m going to use it to bootstrap the build of a Delta style printer that can do larger jobs, over 8 cubic feet.

    I’ve already ordered “The Kraken” (here) a multi-hot-end extruder kit that lets you print four materials in one print job.

    I’m thinking of doing something similar to the 3DR (here) but with a print area big enough to do a volume of 8 cubic feet.

    • Kraken hot end (ordered)
    • Aluminium square tubing for frame (RONA)
    • Bearings (Princess Auto)
    • Fasteners with metric threads (RONA)
    • Threaded rod (RONA)
    • Motor controller board (TBD)
    • Brackets (from my own 3d printer!)

    I’m starting on the design now. Even more work to do now.

  • danw 10:58 pm on 2014-01-05 Permalink | Reply
    Tags: disk,   

    Storage Pod for 2.5″ disks 

    I’m a bit of a storage enthusiast and I’ve been trying to find a reasonable cheap way of bulk loading 2.5″ disks. I’ve been decided to make a SAS disk tray for 2.5″ disks with vertical disks, along the line of the Backblaze storage pod (here)

    I stumbled across some reasonably cheap 2.5″, 24 port backplanes with integrated port multipliers (Supermicro BPN-SAS-216EL1 here) and since they were going for less than $200/ea I got 4 of them.

    Basic plan:

    • Chassis from Protocase compatible with HP rail kits
    • HP rail kit
    • four or five 24 disk backplanes
    • 3x 3-pin chassis fans per backplane
    • Redundant power supply
    • some form of power management board for managing the power supply
    • out of band integration with the SAS expanders on the backplanes
    • SFF-8087 to SFF-8088 adaptor plate
    • some custom wiring to provide enough 4-pin molex plugs to the backplanes (most P/S don’t have enough)
    • 3d printed drive sleds for disks
    • 3d printed slots for disks
    • 3d printed brackets for fans

    Sooo… lots of work to do.

  • danw 8:47 pm on 2013-05-31 Permalink | Reply
    Tags: , tech, tips and tricks   

    Reset password on McData 4500 / DS 24M2 

    I got a few EMC DS 24M2 Fibre Channel switches from ebay (they’re re-badged McData 4500′s). I’ve googled around and everywhere I looked people said it was impossible to reset the password on these (firmware ver 7.x.x) without doing a full factory reset and losing any licensing feature that the previous owner installed. (Although this link has a solution for version 8 or newer firmware)

    But lurking though the switch for a few hours I found the way.. dun dun dun….

    First log in with the secret password ‘neergcm”

    pcpauth ?-help
    pcpauth p 0

    What this does is:

    • ifconfig – set ipaddress to, netmask to, gateway to
    • pcp – enables a bunch of sub commands
    • pcpauth ?-help – shows you the many uses of this tool
    • pcpauth p 0 – resets http access to default (password)

    Now if you log in to the web gui with Administrator/password it will prompt you to enter a new password, then force you to log back in with your new password.


  • danw 10:54 am on 2011-08-16 Permalink | Reply
    Tags: perl, – machine readable solaris sar output 

    I’ve been scripting somewhat with sar on Solaris 10. The major problem is that for some reason there is no flag to output machine readable output (ie. make it easy to import into spreadsheets or other script). The most imporant part was adding a time field to each and every block device to make it much easier to create disk statistics.

    So I threw one together

    #!/usr/bin/perl -w
    use strict;
    #default field delimiter/separator
    my $delim = "|";
    while($_ = shift @ARGV) {
            if($_ =~ m/--delim/) {
                    #change the default field delimter
                    $delim = shift(@ARGV);
            else {
                    die "Usage: sar [-A...] | $0 [--delim seperator]n";
    #preset so we don't get any concat empty val errors
    my $latesttime = "";
    #loop through the sar output
    while(<>) {
            #catch time field of output, remove from line
            if($_ =~ s/^(dd[:]dd[:]dd|Average)//) {
                    $latesttime = $1 . $delim;
            #remove leading and tailing whitespace
            $_ =~ s/(^s+|s+$)//;
            #replace spaces with field delimiter
            $_ =~ s/s+/$delim/g;
            #if the line contains any content, print time field and line
            print $latesttime  . $_ if($_ =~ m/^.+$/);
            print "n";

    In use:

    user@example$ ./  -h
    Usage: sar [-A...] | ./ [--delim seperator]
    user@example$ sar -d | ./ | more

    As you can see, it just formats the sar output to be easily used. It does not remove empty lines or remove annoying psudo block devices (ssd36,h). The default delimiter is a pipe because a comma would interfere with device names with a comma. If you change the delimiter via command line (–delim) be sure to be aware of shell escapes (eg –delim ! not –delim !).

compose new post
next post/next comment
previous post/previous comment
show/hide comments
go to top
go to login
show/hide help
shift + esc