Lone Star PHP 2013 – An introduction to the Secure Software Development Lifecycle.

This June I attended Lone Star PHP 2013 in Dallas, Texas, and presented “An Introduction to the Secure Software Development Lifecycle”. The presentation was an introduction to the Secure Software Development Lifecycle, including requirements and design, development, testing, and acceptance. I also covered topics such as implementing ‘Least Privilege’, ‘Policies and Standards’, and ‘Defensive Coding Practices’.  We also discussed operational aspects and risk mitigation.

Here are the presentation slides and sample code.

SurgeCon 2012 Notes

Mainly for my own reference….

[Keynote]
——————————————–
* godaddy: culture of fear – not open – need to talk about failure
* “choice and voice” driving change
* big data – churn
* experience / availability / performance / scalability / adaptability / security / economy’
* wearable computing = a big thing.  really?
———————————————

 

[Scaling Pinterest]
———————————————
* it will fail, keep it simple
* amazon ec2/s3 worked well
* pro/con limited choice means working within confines/planning
* mysql, very good support from Percona
* memcache + redis
* clustering vs sharding
* few failure modes = simplicity
* failure modes: data rebalance breaks, data corruption replicates, improper balance, data authority failure
* dont approach sharding too early, end up denormalizing, harder to add features
* objects and mappings – object tables and mapping tables, queries are pk lookups or index lookups: no joins
* data does not move
* no schema changes required, index = new table
* Service Oriented Architecture: conn limits, isolate functionality, isolate access (security)
* Scaling the team
* search / feeds / followers <- services
* kafka / hadoop – log every action, snapshot dbs, run analytics.
———————————————

 

[they own the pipes]
———————————————
* round trip latency is the killer
* reduce number of requests to 1
* cert size (1024 vs 2048) and buy a ‘better’ certificate? — less “chained”
* optimize tcp stacks / ack[nowledgement] segments
* dont run or allow ssl2 / old ciphers
* DTLS: SSL w/o the TCP mess – rfc6347
———————————————

 

[arch behind fast dns]
———————————————
* ttfb (time to first byte)
* ttfdnsqr (time to first dns query response)
* many http optimization ignore dns impact
* reduce multiple domains
* dns needs: better.. redundancy, routing, design.
* down/slow server in delegation = bad (need a healthcheck/load balancer for roundtrip banding)
* unicast (bad) vs anycast (good)
* delegation uptime matters
* hot potato routing
* backbones and routing: ospf/igp (link state / metrics) and bgp (distance vector)
* mix igp (ospf) with bgp. ospf floods routes, ibgp stacks adjacencies formed in ospf
* depend on igp route metrics
* data sync and monitoring (same ips in different spaces)
* two networks: anycast facing out towards users, unicast data replication in between the application
* enemy = complexity, avoid multi-level delegation,  pick right ttls, put dns hardware close to userbase
* peeringdb.com
* automated detection, manual changes to route around congestion in asia
* openbfd echo daemon on github
* ecmp: multiple links between routers, load balance at layer 3 level.
———————————————

 

[scaling in the cloud at cost and sla]
———————————————
* make solution as cloud provider agnostic as possible.
* SOA architecture, multi-datacenter, minimal cross-DC traffic
* config mgmt (puppet,chef,etc)
* monitoring (nagious, cacti, graphite, etc) w/ basic alerts (cpu,load,memory,qps,latency)
* service transport (http, thrift-rpc, native: memcahce, redis, rabbitmq)
* load balancers (haproxy, varnish,nginx/apache) and ELB (AWS Specific)
* GSLB global server loadbalancing (between datacenters) route users to nearest DC
* DNS/BGP/Anycast/cookies/user-config
* instance sizing (go 64bit)
* tweak garbage collection and memory settings
* deploys (phased) w/ health-checks
* failure detection, retries, fuzzing, queuing
* tagged deploy units (easy rollbacks)
* keep expensive operations local to region/ datacenter
* nodejs “hiveway” caching proxy + triggered updates (look for BrightTag to open source soon?)
* do aggregate roll-ups at once (1 min, 5  min, 1hr, etc)
*
———————————————

 

[mysteries of a CDN Explained]
———————————————
* find/send user to closest node (beat speed of light)
* ip geotarget / anycast dns
* cdn shield / middle tier between cdn origin and cdn pop
* check out ganglia2 / arista switches
* DSA (non caching cdn / http keepalives voodoo)
* fastly github version of apache
* sysctl values (2 to adjust – ref slides)
* short ttl = survive flash traffic
———————————————

 

==============================================================

 

[who needs clouds? ha in your datacenter]
———————————————
* simple not easy, open not closed, logic not magic
* linux-ha (load balancer)
* IPVS IP Virtual Server
* litmus_paper (open source – github.com/braintree)
* big_brother
* pacemaker
* “thundering heard” and adding resources
* load balancing the load balancers
* OCF (open cluster framework)
* IPaddr2 script
———————————————

 

[realtime web – dirt in production]
———————————————
* find video on web about shouting at disk drives
* voxer PPT app
* DIRTy apps tend to have a human in the loop
* mobile devices at the edge – network transience causes connection state issues.
* illumOS (Joyent)
* Application Restarts = cascading latency bubbles
* if app cant keep up with with tcp backlog, syn packets get dropped
* how to measure / monitor how close we are to the end of the tcp backlog queue?
* github dtrace / tcplistendrop.d
* slow disk i/o – cloud multi-tenancy (running backups, benchmarks) – no insight
* zfsslower.d tool
* dtrace scripts to drill down into particular areas
* heatmap tool (https://github.com/brendangregg/FlameGraph)
* identify latency outliers
* memory growth: whats in the heap? leak (and then where!)? app growth?
* libumem
———————————————

 

[zero to 500k qps – scaling appnexus]
———————————————
* servers + perl + zen = cloud
* auction-based ad-serving
* batch-based change processing
* simple http communication between processes
* packrat log streaming app
* netezza (tupac/icecube), hadoop/hbase (wutag/quest), graphite (real-time)
* netezza vs vertica vs hadoop
* keepalive dns over vendor loadbalancers
* maestro / api driven infrastructure / runbooks
———————————————

 


———————————————
* site architecture:
– load balancer (assigns a webserver)
– webserver (hiphop asssembles data)
– services / cache / databases on the backend
* scaling meta tips
– scale horizontally
– iterate quickly
– ‘gatekeeper’ to stage and rollout code / select what subset of users see
– instrument the world
– “claspin” tool – high-density heatmap viewer for large services, find needle in a haystack, drill down.
– “scuba” tool – in memory datastore – key/value pairs, slice across different sets
– rtwatch tool (realtime watch) – select specific data points, see connections, identify outliers
– ODS tool (operations data store) – site health / key metrics
* scale your tools
* take ownership
* automation: sophisticated systems / failures are common / automate carefully
* faster deployments: central server is bottleneck,  use bittorrent (opentracker)
* challenge: how to restart services quickly enough w/o impacting users
* distributed shell system to run commands on multiple systems at the same time
* fbar tool – facebook auto remediation (write ‘recipes’ or scripts to fix an issue) (think playbooks for NOC engineers)
* fbar uses API + plugins (monitor, config, hardware, etc)
* automation pitfalls: masks systemic problems, cascading failures, unknown actors, cultural fear
* culture wrs: keep teams small, work on most leveraged problem, move people around, constantly prioritize
* scale ops: invest in the people
* ops focus areas: availability, reliability, operation,  efficiency
* roles  to watch out for: the butler, the adversary,  the beautiful mind, the godfather
* fix more, whine less.
———————————————

 

[changing architectural foundation w/ continuous deplyment]
———————————————
* arch needs to change with the business over time
* “boxed” software only gives you 1 chance to deploy
* CI not good for infrq changes /hardware changes/etc
* “culture before tools”
* other people always around to help
* unit tests + functional tests + manual tests
* nagios + naglite2
* “super grep”  -> tail -f | grep
* “deployinator” on github
* graphs: ganglia / graphite
* overlay deployments / code changes with graph stats
* statd (github)
* logster (github)
* “feature flags” allow deploy w/ subset of users to see/use
*  a/b testing for interface changes and to prove interest
* ci pattern: change in small steps.
* dark launch by config. iterate in prod while dark
* maintain old and w in parallel
* ramp up new arch, remove old
* minimize bug hours, trash the schedule, iterate on the tools.
—————————————–

 

[monitoring and debugging nosql inpruio]
———————————————
*blekko  (new search engine)
* conventional monitoring can  too noisy
* monitoring tips: subject = most important  info and body needs enough info to login
* ‘turds’ info in /tmp/ (e.g. output of ps/free/etc)
* be aware that problems clear up
* system hangs destroy evidence on reboot
* wrapper cron jobs
* roll up alerts (ex: 57 alerts that are the same = 1 email)
* “audit” the monitoring system
* statmover.com and saturnalia db
* write your own oom killer and trigger before oom fires
* am i thrashing (swap monitoring)
* identify broken, move out of rotation, flag for follow-up
* automate with scripts common failure recovery tasks
* “beach certify” common tasks

———————————————

 

php://memory Redux

I recently had the opportunity to speak at my local Baltimore PHP User Group and gave a presentation on php://memory and php://temp along with some demos, including one on frequency analysis. Below are the linked code samples and presentation..

php://memory Redux

Code Samples: https://github.com/nanderoo/php-memory-redux

Presenting at PHP|Tek 2012

The schedule is out for the PHP|Tek conference: #Tek12 and I’m excited to announce that I have been selected to give two talks:  Graphing real-time performance with Graphite and php://memory and streams for scaling .

If past years conferences are any indication, like the past tutorials and sessions, this year will only be eclipsed by the un-conference and after-parties. As I have come to learn, the real magic at conferences like this happens in all the space and time between whats on the schedule!

Make sure to follow @mtacon and the #tek12 hashtag on Twitter. There is also a #phptek channel on FreeNode  up around the time of the conference.

Graphing real-time performance Graphite:

Code Samples: https://github.com/nanderoo/php-graphite-demo

PHP Memory and streams:

Code Samples: https://github.com/nanderoo/php-memory-demo

Dreams of disaster at SurgeCon 2011

I recently attended Surge 2011 and thought I would publish some twitter notes and thoughts about the experience. This was the first conference I’ve attended in some time that wasn’t focused on a particular language (like ZendCon) and it was great that Surge was held right here in downtown Baltimore.

Day One

Node js / Chef / Riak

nanderoo Neal Anders Really wish I could be in two places at once for the #riak and #chef training today at #surgecon – until then it’s time to hack some node.js

– riak

rustyio Rusty Klophaus by nanderoo Riak 1.0 shipped! So much good stuff is in this release: blog.basho.com/2011/09/30/Ria… #nosql #basho #riak
nanderoo Neal Anders Great News! Was stuck compiling #erlang RT @seancribbs: @nanderoo We will have pre-built installs for Mac, deb/ubu, centos. #riak #surgecon

– chef

jtimberman Joshua Timberman by nanderoo Chef Repository used in the Chef Training at SurgeCon 2011: github.com/jtimberman/sur… #opschef #surgecon
nanderoo Neal Anders Have to say my favorite session so far today at #surgecon has been the node.js hackathon – bummed I didn’t attend the #riak session.
keithf4 Keith Fiske by nanderoo Can definitely tell who’s tweeting from #surgecon with autocorrect. Lot of “risk” training going on.
nanderoo Neal Anders In the Chef Training (presented by @opscode) at #surgecon
nanderoo Neal Anders #surgecon “your system is running low on virtual memory” twitpic.com/6rydk1
nanderoo Neal Anders @dlutzy @saschabates Would have to check with the #mongodb folks on that – I’ve only seen it partitioned into slices. #surgecon #pizza
nanderoo Neal Anders #surgecon pizza != webscale – should have gone with @nathenharvey for sushi
nanderoo Neal Anders RT: @JonWChicago: LUNCHException: pizza not found. #surgecon yfrog.com/g0x7pzj
nanderoo Neal Anders Pizza and hacking node.js at #surgecon nom nom
nanderoo Neal Anders Anyone know whats on the menu today for lunch at #surgecon ? All this node.js talk is making me hungry. #NoBreakfast

DayTwo

Ben Fried – Keynote Address:

solarce Brandon Burton by nanderoo “The hallmark of a professional is a dedication to self improvement” Ben Fried #surgecon
dje Darrin Eden by nanderoo root cause of scalability failure: cultural #surgecon
richparet Rich Paret by nanderoo Google’s Ben Fried: industrial era notion of org scale via job specialization needs to be discounted. #surgecon
solarce Brandon Burton by nanderoo “cult of devops!!” #surgecon
dje Darrin Eden by nanderoo scalability requires generalists and deep, end-to-end understanding. #surgecon
freire_da_silva Alexandre Freire by nanderoo Root cause of google’s ben fried’s disaster porn: scaling the organization by specialization! Nobody knew the whole system! #surgecon
joshu joshua schachter by nanderoo i am gonna start a CS journal where to get published you have to include your code. because i don’t believe this shit works half the time.
obfuscurity Jason Dixon by nanderoo “If you save it until the last minute you only need a minute to fix it.” #surgecon
obfuscurity Jason Dixon by nanderoo Disaster porn. #surgecon

Artur Bergman – A journey throu the full stack in search of performance and reliability:

solarce Brandon Burton by nanderoo “everything is shit. (but you should make it better)” paraphrasing @crucially #surgecon
solarce Brandon Burton by nanderoo Circular sharding. It’s webscale.
saschabates Sascha by nanderoo Artur Bergman: the most colorful speaker at #surgecon
solarce Brandon Burton by nanderoo If *you* didn’t fix it, it ain’t *fixed*. –@crucially #surgecon
nanderoo Neal Anders Why I attend conferences like #surgecon : @crucially ‘s “full stack” session is all meat and potatoes so far – #deepdive #nofluff #nohype
nanderoo Neal Anders In Artur Bergman: “A journey through the full stack in search of performance and reliability” at #surgecon

Hubert Fonseca and Andre Calvani – Using complex event processing to gather information from infrastructure:

nanderoo Neal Anders In “Using complex event processing to gather information from infrastructure” by Hubert Fonseca and André Galvani at #surgecon

Maxwell Luebbe, Dr. Jia Guo, and Raymind Blum – Google group session:
solarce Brandon Burton by nanderoo “everything is shit. (but you should make it better)” paraphrasing @crucially #surgecon
solarce Brandon Burton by nanderoo Circular sharding. It’s webscale.
saschabates Sascha by nanderoo Artur Bergman: the most colorful speaker at #surgecon
solarce Brandon Burton by nanderoo If *you* didn’t fix it, it ain’t *fixed*. –@crucially #surgecon
nanderoo Neal Anders Why I attend conferences like #surgecon : @crucially ‘s “full stack” session is all meat and potatoes so far – #deepdive #nofluff #nohype
nanderoo Neal Anders In Artur Bergman: “A journey through the full stack in search of performance and reliability” at #surgecon

Robert Treat – Address vendor weaknesses in user-space:

nanderoo Neal Anders Headed over to “Addressing Vendor Weaknesses In User-space” by Robert Treat #surgecon

Panel Discussion – Pushing big data to the cloud:

….??

Day Three

Theo Schlossnagle – Architectures for real-time data:

peschkaj Jeremiah Peschka by nanderoo Fantastic talk by @postwait at #surgecon
nanderoo Neal Anders “write in c, its a cleansing experience” via @postwait #surgecon
nanderoo Neal Anders 350,000 metrics per second – impressive. #surgecon
solarce Brandon Burton by nanderoo “we don’t use Ruby. Our stuff works” @postwait #surgecon
nanderoo Neal Anders “debugging in a distributed system is like playing Russian Roulette” via @postwait #surgecon
solarce Brandon Burton by nanderoo “In “two” months, you may not be able to build the next thing, because you’ll be maintaining the last thing you built” @postwait #surgecon
nanderoo Neal Anders”sharding isnt magic, it is traumatic” #surgecon via @postwait
nanderoo Neal Anders Ready for “Architectures for real-time data” by Theo Schlossnagle but I think my 1st cup of coffee was decaf so I’m only half-here #surgecon

Baron Schwartz – Extracting scalability and performance metrics from TCP traffic:

…?

Mike Panchenko – Building cloud service on a cloud infrastructure:

…?

Wez Furlong – Practical lessons learned in scaling at Message Systems:

chrismunns chrismunns by nanderoo The MessageSystems talk at #surgecon is one of the best. Really interesting architecture. Good job to the speaker
nanderoo Neal Anders “its awesome and its really technical” via @wezfurlong #surgecon
nanderoo Neal Anders Hmnn… “images with a high proportion of skin-tone” #surgecon
nanderoo Neal Anders Post-lunch food coma, just in time for @wezfurlong ‘s “Practical Lessons Learned in Scaling at Message Systems” #surgecon

Rob Cope – Cloudbursting with Amazon EC2 and SQS:

nanderoo Neal Anders “Design for the ‘one in a million’ occurrence.. it happens all the time in the cloud” #cloudbursting #surgecon
nanderoo Neal Anders “SIDS – Sudden Instance Death Syndrome” … “dont even ssh in, just shoot it in the head” #cloudbursting #surgecon
nanderoo Neal Anders Great Tip: Auto Scaling – Set cap to prevent bankruptcy! #cloudbursting #surgecon
nanderoo Neal Anders “Cloudbursting with Amazon EC2 and SQS” by Rod Cope #surgecon

Geir Magnusson – When business models attack:

nanderoo Neal Anders In @geirmagnusson ‘s session – Best slide so far: the one where he showed prod traffic and noted the load testing spike. #testprod #surgecon
thommay Thom May by nanderoo Wondering if @geirmagnusson is going to get a “fn(x) is hiring” line into every slide 🙂 #surgecon
nanderoo Neal Anders “When ever someone uses bullet points in a presentation, a kitten dies.” #surgecon
nanderoo Neal Anders “When Business Models Attack” by Geir Magnusson up next for me at #surgecon
nanderoo Neal Anders Scoring @geirmagnusson ‘s session in my top-3 for #surgecon
papa_fire Leon Fayer by nanderoo One of the best talks of #surgecon by @geirmagnusson to finish it off right! Already looking forward to next year.
nanderoo Neal Anders Does everyone at #surgecon know that @geirmagnusson is hiring? He’s hiring.

– closing session:

nanderoo Neal Anders Closing Session for #surgecon

wezfurlong Wez Furlong by nanderoo Another great #surgecon wrapped up. Props to @OmniTI for gathering a great crowd of tenacious thinkers and doers!

ryancnelson ryan nelson by nanderoo “…point to the place on the doll where the Operating System touched you.” #surgecon

cdferry Chris Ferry by nanderoo So fucking true – “This is why Systems Administrators are angry” – “Packaging other peoples software” #surgecon

Additional Notes:

There where a few articles that came out about the conference

cschammel Chris Schammel by nanderoo Another article about #surgecon on gigaom: goo.gl/cacZg
cschammel Chris Schammel by nanderoo Article on Ben Fried keynote at #surgecon: goo.gl/Pvjiw
keithf4 Keith Fiske by nanderoo Surge 2011 on Slashdot – tech.slashdot.org/story/11/10/01… #surgecon

– no mobile site

– outdated website / breakfas

nanderoo Neal Anders Plenty of coffee and power outlets at #surgecon – no sight of the continental breakfast yet though? #hungry

nanderoo Neal Anders Love the setup for the vendor hall w/ breakfast. Great way to get your munch on and talk w/ folks. #surgecon twitpic.com/6sbzze

– set hard stops for presenters

– links to presenter info (twitter / website)

– video / content online (why not till january?)

– irc channel

nanderoo Neal Anders #surgecon channel up on #freenode for those interested.

– noted tweets:

_tr TR Jordan by nanderoo Give your developers root on production. I think this is the third time I’ve heard this, this time by @beamrider9. #surgecon #gimmeroot
obfuscurity Jason Dixon by nanderoo Another great #surgecon, but what happened to all the failure stories like we had last year?
brennor42 Brennor by nanderoo Sensing some deja vu from yesterday’s sessions… disks suck, networks suck… #surgecon
saschabates Sascha by nanderoo #surgecon emergent theme: complex systems cannot be effectively diagnosed without smart generalists who understand them end to end
cdferry Chris Ferry by nanderoo Push responsibility to the edge. Developers must be on call. Sys Admins should be escalated to. #surgecon
_tr TR Jordan by nanderoo “When the developer deletes the server, what should you do?” “First thing, give them a hug. They probably need it.” #surgecon
dje Darrin Eden by nanderoo If an alert doesn’t have a link directly to a playbook it goes straight to the incident commander. [ed. brilliant!] #surgecon

DenishPatel DenishPatel by nanderoo Interesting method for Software Development ! 1. Build V1 quickly 2. V2 “correctly” /cc @katemats #surgecon
obfuscurity Jason Dixon by nanderoo “If you could do all that why isn’t it automated?” “The short answer is FEAR.” #surgecon
davezwieback Dave Zwieback by nanderoo “Specialization is for insects”. RT @solarce: Was @bfnyc inspired by elise.com/quotes/a/heinl… ? #surgecon

toArray() with Doctrine 2 and Zend Forms.

Based on a couple of assumptions (like ‘NS’ is your library that handles the Doctrine Entity Manager) ..in your abstract class, will need 2 methods:

/**
 *  A way to force eager loading.
 */
public function forceEagerLoad() {
    return true;
}

/**
 * Returns the object and its properties as an array.
 */
public function toArray() {
    $tmpMergedMappings = array();
    $tmpFieldMappings = array();
    $tmpAssocMappings = array();
    if(!$this->em) { $this->em = NS::em(); }
    $testObj = $this->em->find(get_class($this), $this->id);
    $testJob = $testObj->job;
    $tmpFieldMappings = $this->em->getClassMetadata(get_class($this))->fieldMappings;
    $tmpAssocMappings = array_keys($this->em->getClassMetadata(get_class($this))->associationMappings);
    foreach($tmpFieldMappings as $fmKey => $fmValue) {
        if(is_object($this->$fmKey)) {
            if (get_class($this->$fmKey) == "DateTime" ) {
                switch ($tmpFieldMappings[$fmKey]["type"]) {
                    case "sndatetype":
                        $tmpMergedMappings[$fmKey] = $this->$fmKey->format('m/d/Y');
                        break;
                    // handle any custom types..
                    default:
                        $tmpMergedMappings[$fmKey] = $this->$fmKey->format('Y-m-d H:i:s');
                        break;
                }
            } else {
                // presume the default _id mapping...
                $key_id = $fmKey."_id";
                $tmpMergedMappings[$key_id] = $this->$key_id->id;
            }
        } else {
            $tmpMergedMappings[$fmKey] = $this->$fmKey;
        }
    }
    foreach($tmpAssocMappings as $amKey => $amValue) {
        $tmpKey = $amValue."_id";
        switch (get_class($this->$amValue)) {
            case "Doctrine\ORM\PersistentCollection":
                // dont do anything with these right now..
                break;
            default:
                // Trigger the loading via the proxy.
                if(method_exists($this->$amValue, 'forceEagerLoad')) {
                    $forced = $this->$amValue->forceEagerLoad();
                } else {
                    // Note: these classes dont have/inherit a forceEagerLoad() method,
                    // or we are trying to call it on something not set yet.
                    //var_dump(get_class($this->$amValue));
                    //var_dump($amValue);
                }
                if($this->$amValue) {
                    if($this->$amValue->id != null) {
                        $tmpMergedMappings[$tmpKey] = $this->$amValue->id;
                    }
                }
            break;
        }
    }
    return $tmpMergedMappings;

}

..and then in your Zend Controller action, say for editing:

/**
 *
 */
public function editAction() {
    $id = $this->getRequest()->getParam('id');
    $role = $this->em->find('NS\Role', $id);
    if(empty($role)) {
        // handle error
        return $this->_helper->redirector->gotoUrl('/role');
    } else {
        $this->view->role = $role;
        $this->view->form = $this->roleForm($role->toArray());
    }
}

..and your form can look something like this:

/**
*
*/
public function roleForm($data = null) {
    $form = new Zend_Form();
    $form->setAction($this->view->baseUrl().'/role/create')->setMethod('post');
    // id (hidden)
    $id = new Zend_Form_Element_Hidden('id');
    $id->removeDecorator('Label');
    $id->removeDecorator('HtmlTag');
    $form->addElement($id);
    // name
    $name = new Zend_Form_Element_Text('name');
    $name->setLabel('Name');
    $name->setRequired(true)->addValidator('NotEmpty');
    $form->addElement($name);
    // description
    $description = new Zend_Form_Element_Text('description');
    $description->setLabel('Description');
    $description->setRequired(true)->addValidator('NotEmpty');
    $form->addElement($description);
    // submit button
    $submit = new Zend_Form_Element_Submit('Save');
    $form->addElement($submit);
    if($data) {
        $form->setDefaults($data);
        $form->populate($data);
    }
    return $form;
}

Notes from ZendCon 2010 #zendcon #zc10

Brain dumping ZendCon 2010..  If you find any broken links or have links to slides/people I wasn’t able to find, please let me know! I’m aware that some presenters are holding back their slide decks. And some folks (mostly from the IBM-i sessions) don’t seem to have blogs or twitter accounts?

Overall Impressions:

This was my first ZendCon, and my overall impression is a positive one. I walked away with a much better understand of the community and Zend’s involvement in it. I also was very fortunate to meet many new people from all over the world and of wide skillset and experience. The networking and conversations that took place outside of the sessions and at restaurants or poolside over drinks is where the real connections are made.

Some Highlights:

I didn’t know what to think:

  • The food (breakfast and lunch) where about what you would expect from a conference of this caliber. I found myself more than once wanting to leave the venue at lunch and seek real food. Of exception is the dinner provided at the receptions in the evenings.
  • The vendor expo / floor. What a sad turnout (although I was told it was on-par with last years). All you had to do was walk by the Cloud Expo hall to catch a glimpse of what a real show looks like. I was also told the prize/swag ratio was higher at last year’s conference.
  • The constant fawning by some vendors to recruit the attendees. It was like watching a Jr. High School dance. I wish I could have worn a “got telecommute?” shirt. That would have started conversations with companies I’d be interested in.

Big Letdowns:

There were two main low points for me at the conference:

  • The scheduling snafu that caused Jonathan Wage’s sessions to get canceled. One of the main reasons I was looking forward to ZendCon was the sessions on Doctrine2. I’m not sure what lead to this, and I would hope it was a fluke.
  • The Keynotes and ‘The Cloud’. I’ve been to a few internet/tech conferences, and I’d like to think I can recognize when a presentation is not reaching it’s target audience. Most of the keynotes at ZendCon were no exception. If they were related to Zend products, I’d guess most of the attendees didn’t learn anything new. If the presentation at any point used “cloud” more than once, it instantly lost credibility with me (and I was not alone as many users in the #zendcon IRC channel chimed in similar skepticism). The CloudExpo conference was next door and I wondered more than once if the speaker had wandered into the wrong hall.

Books to read:

Completely Random:

  • Robotic Vacuum Overlords (via @naderman)
  • The Nikon to Canon Ratio – why do most php developers prefer Nikon? (the ratio was 5-1 at the conference by my count)
  • Need to follow-up with David Abdemoulaie (@hobodave) – Re: Doctrine2 pagination
  • My Tweets from the week of ZendCon.
  • I need to brush up on my German and French, learn Russian.

My joind.in Comments:

Sessions and Slides:

Nov 01, 2010

Nov 02, 2010

Nov 03, 2010

Nov 04, 2010

Uncons, etc..  ping me if you have more info on these or others:

Getting Doctrine 2 and CodeIgniter 1.7 and PHP 5.3 and MySQL 5.1 and MongoDB 1.4 to play nice – across databases and objects.

Disclaimer: This post covers at a very high level an approach I’m taking into a possible solution to a challenge I’m facing at work. It should in no way be deemed the end-all, be-all, de facto standard. In fact, I’d love to hear your alternative approaches in the comments below!

There are some things that are  not covered in this post, including:

  • How to install/compile/configure PHP, CodeIgniter, Doctrine, MySQL, or MongoDB.
  • The reasons why we are using a Relation Database and a Document-based Database. No really, don’t ask.
  • How to use the command-line features of Doctrine to auto-generate your database schema.

There are some places in the code sample that you will need to edit and modify to your environment  if you copy-paste this code into your own works. Those items are in ALL_CAPS and <<CONTAINED_WITHIN_LT_GT_SIGNS>>. Just plug in your own values for these.

So,

We begin by starting off with a great example from the Doctrine v2 documentation / cookbook: Integrating with CodeIgniter. This will be the basis for our CodeIgniter Library, with a few modifications, including:

Adding in the Doctrine MongoDB ODM namespaces. You’ll noticed that we aliased the MongoDB\Configuration since it clashes with the ORM\Configuration.

<?php
use Doctrine\Common\ClassLoader,
    Doctrine\Common\Annotations\AnnotationReader,
    Doctrine\Common\Cache\ArrayCache,
    Doctrine\ORM\Configuration,
    Doctrine\ORM\EntityManager,
    Doctrine\DBAL\Logging\EchoSQLLogger,
    Doctrine\ODM\MongoDB\DocumentManager,
    Doctrine\ODM\MongoDB\Mongo,
    Doctrine\ODM\MongoDB\Configuration as MongoDBConfiguration,
    Doctrine\ODM\MongoDB\Mapping\Driver\AnnotationDriver;

class Doctrine {

    public $em = null;
    public $dm = null;

    public function __construct()
    {
        // load database configuration from CodeIgniter
        require_once APPPATH.'config/database.php';

        // Set up class loading. You could use different autoloaders, provided by your favorite framework,
        // if you want to.
        require_once APPPATH.'libraries/Doctrine/Common/ClassLoader.php';

        $doctrineClassLoader = new ClassLoader('Doctrine', APPPATH.'libraries');
        $doctrineClassLoader->register();
        $entitiesClassLoader = new ClassLoader('DW', APPPATH.'models');
        $entitiesClassLoader->register();
        $proxiesClassLoader = new ClassLoader('Proxies', APPPATH.'models/proxies');
        $proxiesClassLoader->register();

        // Set up caches
        $config = new Configuration;
        $cache = new ArrayCache;
        $config->setMetadataCacheImpl($cache);
        $config->setQueryCacheImpl($cache);

        // Mapping Configuration
        $driverImpl = $config->newDefaultAnnotationDriver("/<<PATH_TO_WEBSITE_CODE>>/system/application/models");
        $config->setMetadataDriverImpl($driverImpl);

        // Proxy configuration
        $config->setProxyDir('<<PATH_TO_PROXIES>>');
        $config->setProxyNamespace('Proxies');

        // Set up logger
    // commented out for now..
    //  $logger = new EchoSqlLogger;
    //  $config->setSqlLogger($logger);

        $config->setAutoGenerateProxyClasses( TRUE );

        // Database connection information
        $connectionOptions = array(
        'driver' => 'pdo_mysql',
        'user' =>     "<<DB_USERNAME>>",
        'password' => "<<DB_PASSWORD>>",
        'host' =>     "<<DB_HOST>>",
        'dbname' =>   "<<DB_NAME>>"
        );

        // Create EntityManager
        try { $this->em = EntityManager::create($connectionOptions, $config); } catch (Exception $e) { var_dump($e->getMessage()); }

        /**
        * MongoDB handler...
        */
        $configD = new MongoDBConfiguration();
        $configD->setProxyDir('<<PATH_TO_MONGODB_PROXIES>>');
        $configD->setProxyNamespace('Proxies');

        $readerD = new AnnotationReader();
        $readerD->setDefaultAnnotationNamespace('Doctrine\ODM\MongoDB\Mapping\\');
        $configD->setMetadataDriverImpl(new AnnotationDriver($readerD, APPPATH.'models/<<NAMESPACE>>'));

        try { $this->dm = DocumentManager::create(new Mongo("mongodb://<<MONGODB_SERVER>>"), $configD); } catch (Exception $e) { var_dump($e->getMessage()); }

    }

We then add methods to wrap around and make transparent which types of objects we are dealing with. Note that we just touch on some basic functionality such as find() and persist():

public function find($_entity, $_key) {
 $result = null;
 if (property_exists($_entity, '_docORM')) {
 try { $result = $this->dm->find($_entity, $_key); } catch (Exception $e) { var_dump($e->getMessage()); }
 } else {
 try { $result = $this->em->find($_entity, $_key); } catch (Exception $e) { var_dump($e->getMessage()); }
 }
 return $result;
 }

 public function findBy($_entity, $_keys = array()) {

 $result = null;
 if (property_exists($_entity, '_docORM')) {
 try { $results = $this->dm->getRepository($_entity)->findBy($_keys); } catch (Exception $e) { var_dump($e->getMessage()); }
 } else {
 try { $results = $this->em->getRepository($_entity)->findBy($_keys); } catch (Exception $e) { var_dump($e->getMessage()); }
 }
 return $results;
 }

 public function getRepository($_entity) {
 return $this->em->getRepository($_entity);
 }

 public function persist($obj) {
 if (property_exists($obj, '_docORM')) {
 try { $this->dm->persist($obj); } catch (Exception $e) { var_dump($e->getMessage()); }
 } else {
 try { $this->em->persist($obj); } catch (Exception $e) { var_dump($e->getMessage()); }
 }
 }

 public function flush() {
 $this->dm->flush();
 $this->em->flush();
 }

}

You may have noticed that we check to see if a property is set on our object: “_docORM”. This is the flag, a static value we set, in our class that tells use to use the MongoDB ODM calls instead of the Relation Database ORM calls.

Examples of class/Entity might look like this:

A MySQL based class:

<?php
namespace <<NAMESPACE>>;

/**
 * @Entity
 * @Table(name="<<TABLE_NAME>>")
 */
class Role {

 /**
 * @Id
 * @Column(type="integer")
 * @GeneratedValue
 */
 protected $id;

 /** @Column(length=50) */
 protected $name;

 /** @Column(length=255) */
 protected $description;

 /**
 * @ManyToMany(targetEntity="Permission")
 * @JoinTable(name="role_permissions",
 *            joinColumns={@JoinColumn(name="role_id", referencedColumnName="id")},
 *            inverseJoinColumns={@JoinColumn(name="permission_id", referencedColumnName="id")}
 *           )
 */
 protected $permissions = array();

 /**
 * @ManyToMany(targetEntity="Role")
 * @JoinTable(name="role_subrole",
 *            joinColumns={@JoinColumn(name="role_id", referencedColumnName="id")},
 *            inverseJoinColumns={@JoinColumn(name="subrole_id", referencedColumnName="id")}
 *           )
 */
 protected $subroles = array();

 // A way to check for recursive sub-roles
 public function hasSubRoles() {
 if(count($this->subroles) > 0) {
 return true;
 } else {
 return false;
 }
 }
}

A MongoDB based class:

<?php
namespace <<NAMEPSACE>>;

/**
 * @Document(db="<<MONOGO_DB>>", collection="<<MONGO_COLLECTION>>")
 */
class documentRecord {

 static $_docORM = true;

 /**
 * @Id
 */
 protected $id;

 /** @String */
 protected $name;

 /** @String */
 protected $description;
}

Notice too that we are using the docblock annotation to tell Doctrine about relationships and the database/document structure. No need to manage separate YAML or XML files.

Now, from within our CodeIgniter controller, we can transparently interact with database and document-based objects, like so:


function test() {

 $orm = $this->doctrine;
$r = $orm->find('<<NAMESPACE>>\Role', <<ROLE_ID>>);
var_dump($r);

$dr = $orm->find('<<NAMSPACE>>\DocumentRecord', <<DOC_ID>>);
var_dump($dr);

}

So, what do you think?

Notes from PHP-Tek 10 #Tekx

PHP-Tek 10 was organized by PHP Architect and Blue Parabola and held in Chicago, IL from May 18th – 21st this year. This was my first “PHP conference” and I was surprised with the breadth of related topics covered. The was allot of buzz in the air, but a few topics seemed to be very prominent, including the challenges of scalability, nosql and mongodb, hip-hop, application security,  frameworks, and code testing and releasing management .

Below are links to the slides (if the presenters posted them online) and blog and twitter info for each presenter. Some of the presentations I haven’t been able to find online, so if you have the links, send them my way!

Building a Zend Framework application (Rob Allen [Twitter / Blog])

  - Slides: http://akrabat.com/wp-content/uploads/TekX-ZF-Tutorial.pdf

Converting Your MySQL App to NoSQL with MongoDB (Kristina Chodorow [Twitter / Blog])

Bad Guy For a Day A Websecurity hands-on tutorial (Arne Blankerts [Twitter])

   - Slides: http://www.slideshare.net/TheSeer/bad-guy-for-a-day-a-websecurity-handson-tutorial

Javascript for PHP Developers (Ed Finkler [Twitter / Blog])

   - Slides: http://funkatron.com/content/JSforPHPdevs-tekx.pdf

PHP Best Practices (Matthew Weier O’Phinney [Twitter / Blog]) / (Lorna Jane Mitchell [Twitter / Blog])

   - Slides: http://www.slideshare.net/lornajane/best-practices-tekx

PHP Code Review (Sebastian Bergmann [Twitter / Blog]) / (Arne Blankerts [Twitter])

   - Slides: http://www.slideshare.net/sebastian_bergmann/php-code-review-4142719

The Lost Art of Simplicity (Josh Holmes [Twitter / Blog])

   - Slides: http://www.slideshare.net/joshholmes/the-lost-art-of-simplicity

Anti-spam and anti-gaming (Eli White [Twitter / Blog])

- Slides: http://eliw.com/presentations/tek-2010/tek-2010-antispamgame.pdf

Apache Cookbook (Rich Bowen [Twitter / Blog])

- Slides: http://www.slideshare.net/rbowen/apache-cookbook-tekx-chicago-2010

Working with Zend_Form (Rob Allen [Twitter / Blog])

   - Slides: http://akrabat.com/wp-content/uploads/TekX-Zend-Form.pdf

Advanced Date/Time handling with PHP (Derick Rethans [Twitter / Blog])

   - Slides: http://derickrethans.nl/talks/time-tek10.pdf

PHP Essentials (Beth Tucker [Twitter / Blog])

Graphs, Edges & Nodes: Untangling the Social Web (Joël Perras [Twitter / Blog])

   - Slides: http://www.slideshare.net/jperras/graphs-edges-nodes-untangling-the-social-web

Large Scale Systems (David Strauss [Twitter / Blog])

Subversion in a Distributed World (Lorna Jane Mitchell [Twitter / Blog])

   - Slides: http://www.slideshare.net/lornajane/subversion-in-a-distributed-world

Flex + Flickr = Fleckr? Part 1 (Keith Casey [Twitter / Blog])

PHP Looking Into the Future (Scott MacVicar [Twitter / Blog])

  - Slides: http://talks.macvicar.net/tekx-php-future.pdf

Getting Git (Travis Swicegood)

Flex + Flickr = Fleckr? Part 2 (Keith Casey [Twitter / Blog])

SQL Injection Myths and Fallacies (Bill Karwin [Blog])

   - Slides: http://www.slideshare.net/billkarwin/sql-injection-myths-and-fallacies

Code & Release Management (Eli White [Twitter / Blog])

- Slides: http://eliw.com/presentations/tek-2010/tek-2010-coderelease.pdf

Best and Worst Practices Building Rich Internet Applications RIAs (Josh Holmes [Twitter / Blog])

Domain NoSQL: Next Generation Play-Doh (Matthew Weier O’Phinney [Twitter / Blog])

Continuous Inspection and Integration of PHP Projects (Sebastian Bergmann [Twitter / Blog])

  - Slides: http://www.slideshare.net/sebastian_bergmann/continuous-integration-of-php-projects-4159699

Desktop Apps with PHP and Titanium (Ben Ramsey [Twitter / Blog])

   - Slides: http://www.slideshare.net/benramsey/desktop-apps-with-php-and-titanium

10 Developer Trends in 2010 (Matthew Schmidt)

A Web Application Framework for People who Hate Frameworks – Lithium (Nate Abele [Twitter / Blog]) / (Joël Perras [TwitterBlog]) /

- Slides: http://www.slideshare.net/jperras/tekx-a-framework-for-people-who-hate-frameworks-lithium

Introduction to Testing with Selenium (Arne Blankerts [Twitter])

  - Slides: http://www.slideshare.net/TheSeer/intro-toselenium

XDebug (Derick Rethans [Twitter / Blog])

   - Slides: http://derickrethans.nl/talks/xdebug-tek10.pdf

Agile in a waterfall world (Jason Sweat [Twitter / Blog])

   - Slides: http://blog.casey-sweat.us/talks/tekx_AgileWaterfall.pdf

MongoDB for Mobile Applications (Kristina Chodorow [Twitter / Blog])

New SPL Features in PHP 5.3 (Matthew Turland [Twitter / Blog])

   - Slides: http://www.slideshare.net/tobias382/new-spl-features-in-php-53

Streams, Sockets and Filters – Oh My (Elizabeth Marie Smith [Twitter / Blog])

  - Slides: http://elizabethmariesmith.com/slides/Streams,%20Sockets%20and%20Filters%20Oh%20My!.pdf
  - Slides (Notes): http://elizabethmariesmith.com/slides/Streams,%20Sockets%20and%20Filters%20Oh%20My!%20-%20notes.pdf

Measuring Your Code (Nate Abele [Twitter / Blog])

Put down the Superglobals! Secure PHP Development with Inspekt (Ed Finkler [Twitter / Blog])

Models for Hierarchical Data with SQL and PHP (Bill Karwin [Blog])

   - Slides: http://www.slideshare.net/billkarwin/models-for-hierarchical-data

The Art of Message Queues (Mike Willbanks)

Tips & Tricks to get the most of PHP with IIS, Windows, and the Windows Azure Cloud (Sumit Chawla) / (Kanwaljeet Singla)

  - Slides: http://www.slideshare.net/ksingla/how-to-get-the-most-with-windows-and-windows-azure

MySQL Scalability (Ligaya Turmelle [Twitter / Blog])

   - Slides: http://www.slideshare.net/ligaya/mysql-55

Building Real-Time Applications with XMPP (Travis Swicegood)

HipHop for PHP (Scott MacVicar [Twitter / Blog])

Caching with Memcached and APC (Ben Ramsey [Twitter / Blog])

   - Slides: http://www.slideshare.net/benramsey/caching-with-memcached-and-apc

Lean Mean PHP Machine (Jason Austin [Twitter / Blog])

  - Slides: http://www.slideshare.net/jfaustin/lean-mean-php-machine

PHP Inside (Derick Rethans [Twitter / Blog])

   - Slides: http://derickrethans.nl/talks/phpinside-tek10.pdf

Open Source Your Career (Lorna Jane Mitchell [Twitter / Blog])

Turning Numbers into Stories (Ryan Stewart [Blog])

Design Patterns (Jason Sweat [Twitter / Blog])

  - Slides: http://blog.casey-sweat.us/talks/tekx_patterns.pdf

Cross Platform PHP (Elizabeth Marie Smith [Twitter / Blog])

   - Slides: http://elizabethmariesmith.com/slides/Cross%20Platform%20PHP.pdf

Replication with MySQL (Ligaya Turmelle [Twitter / Blog])

   - Slides: http://www.slideshare.net/ligaya/mysql-51-replication

TEK-X on the horizon.

The PHP|Tek Conference is a little over a week away, and I’m excited about the opportunity to attend.  I’m also a little bummed that I’ll miss out on the Tutorial Day — specifically the session on “Converting Your MySQL App to NoSQL with MongoDB“. Can someone take notes for me? ;o)

Here are some of the sessions I’m looking forward to:

I’ll also be posting thoughts on Twitter under the #tekx hashtag. Hope to see you there!