Recent Updates Toggle Comment Threads | Keyboard Shortcuts

  • danw 5:06 pm on 2014-05-30 Permalink | Reply  

    ZFS issues! 

    I’ve been having troubles with writes to my home ZFS storage. Writes seem to block, sometimes for hours, but never generate an error. I don’t get messages in dmesg and the per disk error counters dont increment.

    I went on the zfs mailling list, and first recommendation was to check if the zpool was suspended:

    root@Gangsta:~# echo "::walk spa | ::print spa_t spa_name spa_suspended" | mdb -k 
spa_name = [ "rpool" ]
spa_suspended = 0
spa_name = [ "tank" ]
spa_suspended = 0
root@Gangsta:~#

    Second step was to see where the threads were stopping

    root@Gangsta:~# echo "::stacks -m zfs" | mdb -kw
THREAD           STATE    SOBJ                COUNT
ffffff00171eac40 SLEEP    CV                      2
                 swtch+0x145
                 cv_wait+0x61
                 spa_thread+0x225
                 thread_start+8

ffffff0017796c40 SLEEP    CV                      2
                 swtch+0x145
                 cv_wait+0x61
                 txg_thread_wait+0x5f
                 txg_quiesce_thread+0x94
                 thread_start+8

ffffff001713dc40 SLEEP    CV                      1
                 swtch+0x145
                 cv_timedwait_hires+0xe0
                 cv_timedwait+0x5a
                 arc_reclaim_thread+0x13d
                 thread_start+8

ffffff0017143c40 SLEEP    CV                      1
                 swtch+0x145
                 cv_timedwait_hires+0xe0
                 cv_timedwait+0x5a
                 l2arc_feed_thread+0xa6
                 thread_start+8

ffffff001779fc40 SLEEP    CV                      1
                 swtch+0x145
                 cv_timedwait_hires+0xe0
                 cv_timedwait+0x5a
                 txg_thread_wait+0x7c
                 txg_sync_thread+0x12c
                 thread_start+8

ffffff0017424c40 SLEEP    CV                      1
                 swtch+0x145
                 cv_wait+0x61
                 zio_wait+0x5d
                 dbuf_read+0x1e5
                 dbuf_findbp+0xf8
                 dbuf_hold_impl+0x81
                 dbuf_hold+0x2e
                 dmu_buf_hold+0x8e
                 zap_get_leaf_byblk+0x5c
                 zap_deref_leaf+0x78
                 fzap_update+0x6f
                 zap_update_uint64+0x8e
                 ddt_zap_update+0x78
                 ddt_object_update+0x51
                 ddt_sync_entry+0x237
                 ddt_sync_table+0x9f
                 ddt_sync+0x6b
                 spa_sync+0x3b4
                 txg_sync_thread+0x27b
                 thread_start+8

ffffff0566af1820 ONPROC                     1
                 zio_destroy+0x53
                 zio_wait+0x7e
                 dmu_tx_count_write+0x25a
                 dmu_tx_hold_sa+0x46
                 zfs_write+0x55c
                 fop_write+0x6b
                 write+0x2e2
                 write32+0x22
                 _sys_sysenter_post_swapgs+0x149

root@Gangsta:~#

    Most recently I’ve been asked to:

    echo "::stacks -c spa_sync" | mdb -k

Also can you please include 'zpool iostat tank 1' collected over a this period?

    So here is the layout of the disks

    root@Gangsta:~# zpool status tank
  pool: tank
 state: ONLINE
  scan: scrub repaired 103K in 106h36m with 0 errors on Fri May 23 22:36:19 2014
config:

        NAME         STATE     READ WRITE CKSUM
        tank         ONLINE       0     0     0
          raidz2-0   ONLINE       0     0     0
            c6t16d0  ONLINE       0     0     0
            c6t20d0  ONLINE       0     0     0
            c6t24d0  ONLINE       0     0     0
            c6t28d0  ONLINE       0     0     0
            c6t55d0  ONLINE       0     0     0
            c6t53d0  ONLINE       0     0     0
            c6t49d0  ONLINE       0     0     0
            c6t51d0  ONLINE       0     0     0
            c6t54d0  ONLINE       0     0     0
            c6t56d0  ONLINE       0     0     0
            c6t52d0  ONLINE       0     0     0
            c6t50d0  ONLINE       0     0     0
        logs
          c6t26d0s0  ONLINE       0     0     0
          c6t30d0s0  ONLINE       0     0     0
        cache
          c6t26d0s1  ONLINE       0     0     0
          c6t30d0s1  ONLINE       0     0     0
        spares
          c6t18d0    AVAIL   
          c6t22d0    AVAIL   

errors: No known data errors
root@Gangsta:~#

    Output from iostat
    problem.iostat

    Output from mdb
    problem.mdb

    Output from zpool iostat
    problem.zpooliostat

     

  • danw 6:22 pm on 2014-02-03 Permalink | Reply
    Tags: ,   

    3D printer arrived 

    The nifty 3d printer from DIY Tech Shop finally arrived! It’s been a long row to hoe.

    I originally ordered from MakiBox last September (2013), but they are/were having serious production issues; if you want to talk to some seriously annoyed people check out the forum. After the Nth delay (N is >= 4) in mid December 2013 I gave up on MakiBox and cancelled my order, even after they contacted me an assured me that my order was going to be “coming real soon now.” I ordered from DIY Tech Shop, although their website stated that they had run out of parts, and were planning on a large buy in early/mid February 2014. It arrived on Tuesday January 28th, 2014. As of this writing, Monday February 3rd, 2014, MakiBox is still only showing 504 shipped units. I am sure the forum has even more angry people asking the operator of MakiBox to remove his wildly optimistic “shipping estimates”, where by “shipping” they mean “lie” and by “estimates” they mean “LIE!”.

    So it arrived and I’ve gotten the hardware assembly done (see below).

    My oberservation:

    • The brackets for the Z-axis motors were not designed for the shape of these motors, lacking clearance for a ring near the output shaft. The first bracket cracked before I noticed the issue, the second one I shaved by hand to fit the motor
    • A few of the brackets pay more attention to aesthetics than function, lacking some more useful functionality. E.g. left and right sided parts that are functionally interchangeable, but would benefit from having nuts on either side to better fine tune connections.
    • It looks like the threaded rod was not cut with a chop saw, and one end of one of the rods wouldn’t let a nut on. I cleaned it up with a die, but I probably could have made do by spinning a nut down the rod from the good end and using a jam-nut to force it through to clean up the threads.
    • The website advertises a “RAMPS 1.4″ Arduino shield to manage the printer, but the manual specifies annother name. Although aparently they are the same board it confused me, in addition the RAMPS 1.4 board seems to be a dual hot-end controller and the manual doesn’t specify which parts are for the first hot-end and which are for the second hot-end. And since I only have ONE hot-end…..

    Now I need to start on the electronics. It’s just a bunch of crimp-on butt-joints but I’m not super confident of the vary non-specific nature of the instructions.

    3d-printer-semi-complete

    ToDo:

    • Finish wiring
    • Get filiment
    • Print enough replacement brackets for a whole printer
    • Get components for a second hot-end
    • Redesign to a Bowden type extruder
    • Start work on a 3d printer that can print parts at least as large as the width of a rack server
    • Design brackets for the backplanes
     

    • Kevtronic 1:23 pm on 2014-02-05 Permalink | Reply

      Uhhh.. huh huh… you said crimp-on butt-joints… Uhhhh…. huh huh….

    • WiggleChampion 1:46 pm on 2014-02-05 Permalink | Reply

      cool. very very cool.

  • danw 8:00 pm on 2014-01-09 Permalink | Reply
    Tags: ,   

    Storage pod backplanes arrived 

    So the backplanes arrived. They look good. Upon investigation, it turns out these have SAS1 (3gbps) SAS expanders, and can only recognize up to 2048GB of disk, so when 3TB SFF drive appear on the market they will not be able to accommodate them.

    Now I need to make a CAD representation of the PCB so I can design the bracket I want to 3D print to hold the backplane, drives and fans. Hopefully I can do a reasonable unit with easy removal for on site servicing.

    front of backplane

    Front of backplane


    back of backplane

    Back of backplane


    Apparently my work BB takes poor pictures.

     

  • danw 4:16 pm on 2014-01-06 Permalink | Reply
    Tags: , ,   

    Current storage setup 

    I’ve sunk a lot of $$$ into storage over the years and here is my current setup.

    • IBM x3650
    • 12GB ram
    • 2x dual core 3.0Ghz Xeon
    • 2x Fibre Channel PCIe cards
    • LSI SAS HBA
    • 5x SGI SE3016 disk trays (16 disks, 3u rack mount) (here)
    • 64x 40GB SATA disks for high performance I/O
    • 14x 2TB WD RED disks for bulk storage
    • 2x 120GB SSD disks for ZFS cache

    This allows me to host 20+TB of long term storage on the bulk storage, and 2TB for performance I/O, and export ZFS duplicated LUNs via redundant fibre channel switches.

    Future major additions are:

    1. Modify SE3016 disk trays to use SAS2 expanders that support 3TB and larger disks
    2. Attach bulk 2.5″ SFF SSD and SAS disks to storage server via new custom made disk tray (here)
     

  • danw 3:25 pm on 2014-01-06 Permalink | Reply
    Tags: ,   

    starting a delta type 3d printer 

    I’ve ordered a 3d printer (here) and I’ve decided I’m going to use it to bootstrap the build of a Delta style printer that can do larger jobs, over 8 cubic feet.

    I’ve already ordered “The Kraken” (here) a multi-hot-end extruder kit that lets you print four materials in one print job.

    I’m thinking of doing something similar to the 3DR (here) but with a print area big enough to do a volume of 8 cubic feet.

    • Kraken hot end (ordered)
    • Aluminium square tubing for frame (RONA)
    • Bearings (Princess Auto)
    • Fasteners with metric threads (RONA)
    • Threaded rod (RONA)
    • Motor controller board (TBD)
    • Brackets (from my own 3d printer!)

    I’m starting on the design now. Even more work to do now.

     

  • danw 10:58 pm on 2014-01-05 Permalink | Reply
    Tags: disk,   

    Storage Pod for 2.5″ disks 

    I’m a bit of a storage enthusiast and I’ve been trying to find a reasonable cheap way of bulk loading 2.5″ disks. I’ve been decided to make a SAS disk tray for 2.5″ disks with vertical disks, along the line of the Backblaze storage pod (here)

    I stumbled across some reasonably cheap 2.5″, 24 port backplanes with integrated port multipliers (Supermicro BPN-SAS-216EL1 here) and since they were going for less than $200/ea I got 4 of them.

    Basic plan:

    • Chassis from Protocase compatible with HP rail kits
    • HP rail kit
    • four or five 24 disk backplanes
    • 3x 3-pin chassis fans per backplane
    • Redundant power supply
    • some form of power management board for managing the power supply
    • out of band integration with the SAS expanders on the backplanes
    • SFF-8087 to SFF-8088 adaptor plate
    • some custom wiring to provide enough 4-pin molex plugs to the backplanes (most P/S don’t have enough)
    • 3d printed drive sleds for disks
    • 3d printed slots for disks
    • 3d printed brackets for fans

    Sooo… lots of work to do.

     

  • danw 8:47 pm on 2013-05-31 Permalink | Reply
    Tags: , tech, tips and tricks   

    Reset password on McData 4500 / DS 24M2 

    I got a few EMC DS 24M2 Fibre Channel switches from ebay (they’re re-badged McData 4500′s). I’ve googled around and everywhere I looked people said it was impossible to reset the password on these (firmware ver 7.x.x) without doing a full factory reset and losing any licensing feature that the previous owner installed. (Although this link has a solution for version 8 or newer firmware)

    But lurking though the switch for a few hours I found the way.. dun dun dun….

    First log in with the secret password ‘neergcm”

    ifconfig 192.168.0.5 255.255.255.0 192.168.0.1
    pcp
    pcpauth ?-help
    pcpauth p 0

    What this does is:

    • ifconfig 192.168.0.5 255.255.255.0 192.168.0.1 – set ipaddress to 192.168.0.5, netmask to 255.255.255.0, gateway to 192.168.0.1
    • pcp – enables a bunch of sub commands
    • pcpauth ?-help – shows you the many uses of this tool
    • pcpauth p 0 – resets http access to default (password)

    Now if you log in to the web gui with Administrator/password it will prompt you to enter a new password, then force you to log back in with your new password.

    IT WORKS!

     

  • danw 10:54 am on 2011-08-16 Permalink | Reply
    Tags: perl,   

    sarpipe.pl – machine readable solaris sar output 

    I’ve been scripting somewhat with sar on Solaris 10. The major problem is that for some reason there is no flag to output machine readable output (ie. make it easy to import into spreadsheets or other script). The most imporant part was adding a time field to each and every block device to make it much easier to create disk statistics.

    So I threw one together sarpipe.pl:

    #!/usr/bin/perl -w
use strict;

#default field delimiter/separator
my $delim = "|";

while($_ = shift @ARGV) {
        if($_ =~ m/--delim/) {
                #change the default field delimter
                $delim = shift(@ARGV);
        }
        else {
                die "Usage: sar [-A...] | $0 [--delim seperator]n";
        }
}

#preset so we don't get any concat empty val errors
my $latesttime = "";
#loop through the sar output
while(<>) {
        chomp;
        #catch time field of output, remove from line
        if($_ =~ s/^(dd[:]dd[:]dd|Average)//) {
                $latesttime = $1 . $delim;
        }
        #remove leading and tailing whitespace
        $_ =~ s/(^s+|s+$)//;
        #replace spaces with field delimiter
        $_ =~ s/s+/$delim/g;
        #if the line contains any content, print time field and line
        print $latesttime  . $_ if($_ =~ m/^.+$/);
        print "n";
}

    In use:

    user@example$ ./sarpipe.pl  -h
Usage: sar [-A...] | ./sarpipe.pl [--delim seperator]
user@example$ sar -d | ./sarpipe.pl | more

SunOS|bcaeao|5.10|Generic_144488-10|sun4u|08/16/2011

00:00:00|device|%busy|avque|r+w/s|blks/s|avwait|avserv

00:10:01|md110|0|0.0|0|1|0.0|29.9
00:10:01|md111|0|0.0|0|0|0.0|0.0
00:10:01|md115|1|0.0|2|30|0.0|15.3
00:10:01|md116|0|0.0|0|0|0.0|12.8
00:10:01|md120|0|0.0|0|1|0.0|27.4
00:10:01|md121|0|0.0|0|0|0.0|0.0
00:10:01|md125|1|0.0|2|30|0.0|13.4
00:10:01|md126|0|0.0|0|0|0.0|13.0
00:10:01|md130|0|0.0|0|1|0.0|0.0
...
Average|ssd35,c|0|0.0|0|0|0.0|0.0
Average|ssd35,g|1|0.0|2|179|0.0|10.1
Average|ssd36|0|0.0|0|0|0.0|0.0
Average|ssd36,a|0|0.0|0|0|0.0|0.0
Average|ssd36,b|0|0.0|0|0|0.0|0.0
Average|ssd36,c|0|0.0|0|0|0.0|0.0
Average|ssd36,f|0|0.0|0|0|0.0|0.0
Average|ssd36,g|0|0.0|0|0|0.0|0.0
Average|ssd36,h|0|0.0|0|0|0.0|0.0
Average|ssd38|0|0.0|1|5|0.0|2.2
Average|ssd38,c|0|0.0|0|0|0.0|0.0
Average|ssd38,g|0|0.0|1|5|0.0|2.2
user@example$

    As you can see, it just formats the sar output to be easily used. It does not remove empty lines or remove annoying psudo block devices (ssd36,h). The default delimiter is a pipe because a comma would interfere with device names with a comma. If you change the delimiter via command line (–delim) be sure to be aware of shell escapes (eg –delim ! not –delim !).

     

  • danw 11:37 am on 2011-07-13 Permalink | Reply
    Tags:   

    Solaris is cool and stuff 

    While research a problem with backing up Solaris zones I stumbled across a question about Solaris on Serverfault.com.

    And since all the answers didn’t really seem all that good I decided to drop some knowledge:

    This question is funny; this is almost the perfect question for a shill to ask to highlight Solaris 10 new features but no one gave the pro-Solaris answer.

    This is a textbook application of Solaris Zones. The shared kernel provided by Zones lowers the overhead of virtualization, and increases speed dramatically. If you have an idea of a standard install for VPS (bash, apache2, php5, python 2.X, …) you can create a single “gold” zone to use as a template to clone to new zones. Package repositories are available at sunfreeware and blastwave providing you with pre-compiled packages, removing the need to compile your own if you don’t want to.

    You can create your template, charge $X per VPS and clone the template for each new customer, total config time upwards of 5min, 0min if you script/automate it. Upgrading the “global” zone (the base system) will cascade those upgrades into the zones, or you can upgrade per zone, also highly automatable.

    Solaris has kernel space accelerated SSL encryption for supported hardware: expensive cards, Sun/Oracle Niagra2 CPU based systems, and the new Nahalem systems with AES acceleration, which greatly increases the number of SSL protected websites you can host per system (a href=”http://www.c0t0d0s0.org/archives/6926-Performance-Impact-of-kssl.html&#8221; target=”_blank”>link).

    Solaris 10 has many new features in resource management allowing you to segregate individual zones/processes/groups/users and keep runaway or compromised applications in one zone/group/user from impacting any others, as well as all the normal POSIX resource controls on memory use, file descriptors, etc.

    Solaris 10 Zones (and Solaris 10 in general) was designed from the ground up to prove excellent security, accountability, resource management, and to dovetail nicely with Sun (and now Oracle) hardware offerings. When released the Sun T5240 + Sun Solaris + Solaris Zones package was the best platform for page views per second for the money.

    In terms of technical merits, Solaris Zones is probably the best VPS solution available. But as is usually the case the issue is requirements and costs. Licensing, support costs, and Niagra2 or newer CPU hardware costs are rising with the Oracle takeover.

    So evaluate the following: Will the higher VPS density, better VPS isolation and wiz-bang features compensate for higher licensing costs (if using Oracle Solaris), smaller user base to draw peer support from, higher hardware costs (for SSL accel), cost of supporting yet-another-OS, cost of hiring people to support yet-another-OS, the longer time it takes for security patches to get released.

    If you already have a windows team, do you really want to hire a Solaris team just to shave a few percent off of your hardware bill? Stick with Hyper-V until it’ll save you money to switch. If you already have a large deployment of Solaris systems then go with Solaris. If you have a large Linux skill pool to draw on, do a Solaris trial and see how much extra time it takes 3 admins to learn the differences and maintain a new environment for 6 months

    But technology should almost never dictate your business decision process. Much as I hate to say it for most service providers it makes more sense to provide a Windows based VPS system than a Solaris one. Unless you know now that you’re going to need the feature set, and the advantages are going to save you lots of Time And Money(TM) you probably don’t want Solaris.

    But if this isn’t for a business and more about having fun, then go ahead, use Solaris! It’s alot of fun, has tons of features and options that you’ve never even thought of if you’re coming from a non-commercial Unix background. The deeper you get in to Solaris the more you learn about smart engineering and new ways of solving technical problems. I’ve yet to see a Linux box with a “load average: 1000.0+, 1000.0+, 1000.0+” that was responsive and easy to recover.

    @symcbean: I know Solaris (or Slow-laris as it is sometimes called) has a reputation for poor performance (eg your fork example) but I seem to recall that the “Solaris Internals” book said that they re-engineered the threads significantly for Solaris 10, and process creation/forking performance was among the industry leaders. The LWP framework where each thread in an app is mapped to it’s own light weight process in kernel space apparently gave a big boost to performance, reliability, and accounting. The big hurdles for Solaris aren’t so much technical as operational (bad ui), cultural (small user base), and political (Oracle).

    Link to original

     

  • danw 5:10 pm on 2010-06-01 Permalink | Reply
    Tags: solaris liveupgrade   

    Solaris Live Upgrade: NOOOOOOOOO! 

    I’ve been having a lot of fun with Solaris Live Upgrade at work lately. I’ve discovered a few interesting things that I thought I should share.

    Live Upgrade can down your server if you’re not carefull

    I don’t know why, but I’ve managed to down my server twice in the last week trying to create live upgrade boot environments. One zone lost the ability to see any mounted directories there by scaring the crap outta the DBA and requiring a zone reboot to fix, another abbandonned a cpio process copying data to the root file system. While it didn’t cause a crash, it could have broken some processes and SMF requires free space in /etc to operate correctly (aka save crashed processes)

    Live Upgrade lucreate doesn’t fail cleanly

    If live upgrades lucreate fails for any reason, it is very hard to recover. You can’t unconfigure the new boot environment, you cant delete the new boot environment, you can only complete the new boot environment, and that doesn’t work if say there isn’t enough physical space, or another hardware problem emerges

    Live Upgrade ludelete doesn’t work most of the time

    If you accidentaly destroy the metadevices or zfs file systems that live upgrade expects to exist in a boot environment, you cannot delete it. If the boot environment is “in complete” you can’t delete it, you preaty much can’t do anything with ludelete except remove pristine live upgrade environments. AKA only about 10% of the boot environments you wanted to delete.

    Live Upgrade is iffy at best

    So far this week I’ve had live upgrade refuse to patch zones because a single temp file didn’t copy correctly during boot creation. I’ve had lucreate mangle zone names then complain that the mangled name doesn’t exist. If you have a zone that mounts a file system, you have to include it in an exclude list file, or live upgrade will try to copy the contents of that additional file system onto your zones root drive.

    How I Live upgrade

    1. tar up /etc/lu*
    2. ls -al /tmp for each zone including global-zone
    3. create an exclude file listing all that you dont want on the zone’s root filesystem
    4. create a new live upgrade boot environment
    5. luupgrade either to a new version of solaris or to install patches
    6. luactive your new boot environment
    7. reboot as instructed via init 6
    8. confirm that the correct disk/filesystem is booting
    9. delete /etc/lu*
    10. restore /etc/lu* from the tar file
    11. If at any time something doesn’t work right, blow away all of the live upgrade configuration and restore with the tar file. Also remove any additional files from zones under /tmp that may be associated with live upgrade

    This is probably quite bad advice but I find live upgrade only seems to work when it is the first time you are using it. All subsequent times you get stuck with missing file systems that you removed since the last time you upgraded, weird file access errors, zone misnames, file systems filled to 100% with data you didn’t want copied, left of processes changing things you probably didn’t want it to.

     

← Older posts

c
compose new post
j
next post/next comment
k
previous post/previous comment
r
reply
e
edit
o
show/hide comments
t
go to top
l
go to login
h
show/hide help
shift + esc
cancel