Nanoscale electronics, pervasive connectivity, and cloud computing have together ushered in the Internet of Things (IoT). The first generation of IoT has led to myriads of sensors in embedded, wearable, and mobile platforms collecting data to sense the state of various natural, engineered, and human systems with sensory information flowing through distributed multi-tiered networks and distributed computing architectures. A salient feature of IoT has been that connectivity between embedded devices and cloud services has revolutionized sensing systems. The ease with which sensor measurements and commands can flow between the sensors and the cloud allow sophisticated algorithms, massive computing resources, and large-scale data to be brought to bear on the task of sensemaking in multiple domains such as health, energy management, transportation, etc. However, the first generation of sensing-focused IoT systems are precursors of a new generation of IoT systems where the sensor data will be used to influence and control the state of human-cyber-physical systems at multiple scales ranging from personal to societal. The sensor data, instead of being ingested primarily for slower time-scale knowledge discovery and decision making, is becoming part of a complex web of distributed autonomous and semi-autonomous feedback loops controlling and coordinating swarms of autonomous devices owned and managed by multiple parties and intelligently operating in shared spaces while interacting with humans and the physical world around them. This emerging paradigm of pervasive perception, cognition, and action presents a broad spectrum of unprecedented challenges: extreme scale, unstable dynamics, variability and heterogeneity, time and location awareness, ultra-low latency requirements, intermittent resource availability, fragility to attacks and privacy risks. Our current information technology infrastructure – comprising the three distinct and largely independently developed technologies behind the Internet, datacenters, and embedded edge devices – can only suboptimally cope with these challenges. Addressing them requires a distributed computing substrate that takes an integrated view of the key functions - sensing, processing, learning, memory, dissemination, actuation - while optimizing across layers of processing and networking to achieve performance, security, and other guarantees. In this talk I would seek to highlight various challenges and opportunities in the path towards developing an architecture for a networked and distributed computing substrate upon which future applications with perception-cognitionaction loops at extreme and diverse spatiotemporal scales can be hosted with performance, security, robustness, and privacy guarantees. In particular, the talk will described recent work towards (i) A distributed system architecture which supports the new notion of Quality of Time which makes uncertainty in time observable and controllable in order to robustly support time-aware applications across the edge-middle-cloud tiers, (ii) Learning-enabled edge devices with efficient rich inferencing and data-driven modeling for enhancing performance, human-awareness, and security.